Cross-Platform Libraries with Rust

When we create a new release of Oso, we run 87 different Github actions. Those actions build libraries for 6 different languages and test them on different versions of the language across 3 different operating systems. Oso has libraries for Python, Ruby, Java, Javascript, Go, and Rust, and each of those libraries works on Linux, Windows, and macOS. Every library is based on a core that is written in Rust. That allows us to support so many languages without rewriting every feature 6 times. Here's how it works.

Our library: Oso

Oso is an authorization framework that runs inside your application. It comes as a library that you call into to make authorization decisions. Within that library, there is an interpreter for a declarative programming language called Polar. Polar is a language used for writing authorization logic, and it has a lot of nice features that make it easy to interact with your application data.

The first version of Oso was a Python library and the first version of Polar was written in Python. Since we were using the Polar language from within Python, it was very natural to pass Python objects as arguments to Polar queries. It was also easy to look up properties on a Python object or call a method on an object right from the policy. This made it feel like the policy was executing “inside” of your app, rather than requiring you to serialize all the data and send it to the policy. It made the Polar language feel like an extension of Python itself.

This was a very natural way of expressing authorization and our early users liked it. But, we wanted it to work for anyone in any language. We knew we didn't want to port the whole codebase to every language we want to support and that we wanted to keep Oso as a library that worked with application data in the same way. Our best option would be to take the core code and port it to something that could be embedded in all the language libraries.

Multi-language support: moving the core to Rust

We pulled out what we call the "Core" which is the Polar language and policy evaluation code. That way we can develop it as a single codebase and embed it into our language-specific libraries.

We chose Rust for this because we like it, but it also is easy to embed in other languages. Rust code can be compiled down to a native library and can expose a C-compatible API. This means anyone that can talk to a C library can talk to our Rust library, and everyone can talk to a C library.

C is the language the operating system uses to expose its API. If your language can’t talk to C, it can’t do anything useful like open files, allocate memory, or print to the console. This means that every language out there has built-in ways to call C code, so our C-compatible API would work for every language we wanted to support.

Designing the API

Another important goal was to keep the intuitive feel of the Python implementation, especially being able to operate on the host language’s objects. We wanted to be able to look up properties on objects, call host language methods, and detect the classes of objects from Polar. This was all easy to implement when everything was Python, but now we needed some way to resolve these questions while the Rust code was evaluating.

Running a Polar query in the Rust code is like running a script in any other language. You pass in a program, the Rust code evaluates it and returns the answer. The evaluation is a stateful process that needs to pull in information from the host language as it processes. There are a few options for how the host language could communicate with this evaluation.

The first is to just directly check properties or call methods. This is how the first Oso Python version did things, but couldn't work when we moved cross-platform. If you're operating in Rust, even if you have some sort of pointer to a Python object, you can't call a method on it or look up a property without using special Python runtime code. That runtime code would look very different in Python than in JavaScript, for example, and we'd have to write and update that runtime code for every language. Instead, we wanted a more generic way to communicate with the host language while the Rust code is running.

The next option would be some sort of callback. The host language could have a “call a method” callback and a “lookup a property” callback. Then Rust could call those callbacks when it needs to do one of those operations, get the result and continue processing. This is a common style of API in lots of languages, but it doesn’t work here. A function in Java or JavaScript or Python is not just a pointer to a specific instruction like it is in C or Rust. It’s a much more complicated thing and depends on the language runtime. For the same reason we can't just look up properties on a Python or Ruby object we also can't just call a Ruby or Python or Java function in any standard way.

A third option could be to run Polar as a separate process, or on a separate thread and communicate with the host language over a socket or some other kind of channel. This could work, but would make Oso less portable. If we did that, we'd be restricted to environments that had threads or sockets. We want Oso to run everywhere, and a setup like this wouldn’t work in the browser or in embedded scenarios. It’s also actually a lot more complicated than what we decided on.

If Rust can’t easily call back to the host language, then we have to have the host language call Rust. By inverting the control so that everything is driven by the host language, we don’t have to deal with any tricky runtime or callback problems. That way, we can keep our C API as simple functions that take and return values.

This is how it works:

Polar query evaluation is centered around an event loop that is driven by the host. When we run a query we create a new Query object. This is a virtual machine that processes polar queries. The host then calls a next_event function. We run the Polar query until we need something from the host, like the result of a method call. It then pauses the virtual machine and returns an event that says “call this method”. The host does whatever it has to do to process the event, in this case calling the method and sending the result. Then the host calls next_event again and the virtual machine picks up where it left off. When the final Done event comes through, the query is over.

There are events for everything the core needs to talk to the host language about. Method calls, property lookups, results, checking the type of values. These events are JSON strings because nearly every language can easily work with JSON.

Building the Libraries

Now that we have a language that any other language can talk to and an API any language can call we have to figure out how to embed it. The Rust core code also has to be available at runtime so the host language can call into it. We have to build the Rust core into a native library and embed it in the host library somehow. The way we do that depends on the language.

In Java and Ruby, we build the core into a dynamic library and embed it within the package (a Gem for Ruby and a Jar for Java). It’s a dynamic library because we can't link it directly into the Ruby process or the JVM, but we want to be able to call into it at runtime. Dynamic libraries have to be in a different format for each operating system, so we build three different ones: one for macOS, one for Linux, and one for Windows, and include them all in the package. The code then picks the correct dynamic library to talk to at runtime based on the current operating system. This way we have one Oso package that works on any operating system.

In Python, we do something similar. We build Oso into a dynamic library and include it in a wheel package (Python's packaging system for binaries). In Python, however, we can build different packages for each operating system. Pip, the Python package manager, will take care of downloading the correct one when someone installs the package. This is nice because each package only contains the library it needs, which makes it much smaller than the Ruby and Java libraries.

The Javascript package works a little differently. For NodeJS we could have built a dynamic library and called it the same way we do for the others, but we also wanted our library to work in the browser. Native code doesn't run in the browser, but what we can do is compile to WASM, which is a new-ish binary-ish format made for the web. It’s sort of like a native library, but in a new standardized format and will work in multiple JavaScript runtimes and on multiple operating systems. This is pretty cool because now we only have to build one library for JavaScript. Our design decision to not depend on anything other than the host calling us via a simple C API also helped out here—there are no files, sockets, or threads in WASM.

The next language we ported to was Rust. The Rust port was complicated in some ways—for examples of where that got tricky, see the blog posts on the class system we had to build.  But using the core from Rust was very simple. The core is written in Rust, we’re calling it from Rust, so we distribute it as a Rust library. No need to build it beforehand and embed it.

Golang works a little bit differently than the others. Go is a compiled language, and generates an executable binary. That binary can be moved around and run without having the source code available. If we did a dynamic library like some of the other languages, then the user would have to copy that dynamic library around with the binary, which would be extremely annoying. Instead, we build a static library that gets linked into the executable with the Go code. We still need to build that static library for each operating system, but only one will get linked into the compiled binary.

This style of embedding works well and lets us make the Oso library very portable, but there are a few downsides. Not only do we have to build code for each operating system, but we also have to build it for each CPU architecture. Currently, we only support x64 because up until a couple of years ago that was what the vast majority of personal and server computers were using. Recently ARM has gotten a lot more popular, and soon we'll support that too. That will mean that we have to double the number of builds we do.

I've glossed over a lot of the complexity in building and distributing native code. There are a lot of subtle and tedious details that come up in building cross-platform libraries. Deciding between dynamic and static linking of the C runtime library, deciding between Musl and GlibC, deciding between Windows MSVC and Windows GNU—each of these is a decision that complicates our implementation. We’ve had to figure out most of this via trial and error. It's worth it to us because we’re committed to making Oso work for everyone. We’ve continued to fix things as they come up, expand our platform support, and add more and more test platforms to our CI job.

The Future

All of this is still less work than porting features to six versions of the same codebase. There are alternatives that might give us the same shared code advantages but make some of the library complexity easier. One option we’ve been talking about is building an Oso compiler instead of an interpreter. This would be a tool that compiled your policy to some host language code for evaluation. At runtime, there wouldn't be any more Rust code, just generated evaluation code. We'd only have to support every language instead of every language/OS/toolchain/libc/CPU combo. We'd still have a cross-platform compiler binary, but that's a bit easier to manage than cross-platform libraries and would be a build dependency instead of a runtime one.

We're still in the early stages of thinking about that but for now, our native library approach has worked well so far. What languages or platforms would you like to see Oso come to? Let us know on GitHub!

And if you found these problems interesting, you might be interested to know that we are hiring! Or if you'd just like to talk with us a little more, come and join the Oso community.

‍

‍