I'm very much a fan of the idea that language features — and especially library features — should not have privileged access to the compiler.
Rust is generally pretty good at this, unlike (say) Go: most functionality is implemented as part of the standard library, and if I want to write my own `Vec` then (for the most part) I can. Some standard library code relies on compiler features that haven't been marked stable, which is occasionally frustrating, but the nightly compiler will let me use them if I really want to (most of the time I don't). Whereas in Go, I can't implement an equivalent to a goroutine. And even iterating over a container was "special" until generics came along.
This article was a really interesting look at where all that breaks down. There's obviously a trade-off between having to maintain all the plumbing as user-visible and therefore stable vs purely magic and able to be changed so long as you don't break the side effects. I think Rust manages to drive a fairly good compromise in allowing library implementations of core functionality while not needing to stabilise everything before releasing anything.
> I'm very much a fan of the idea that language features — and especially library features — should not have privileged access to the compiler.
At some point I realized I was in the opposite camp and nothing I have seen since that has really changed my view point.
Languages: compilers, libraries, toolkits, etc, aren't supposed to be some abstract collection of parts that can be theoretically hooked together in any possible way to achieve any possible result, they're for solving problems.
You can argue that these things are not opposites, and in theory that's true, but in practice, they seem to be! Go is a good example of making compromises that limit flexibility for the sake of developer/designer convenience.
An interesting example would be Lego, I'd argue that Go is closer to Lego design because it has a bunch of specific pieces that fit together but only in the way the designer intended.
I suspect someone taking the opposite approach from, say, Rust, would argue that some Go pieces don't actually fit together in the way that we think all lego pieces should.
My counter argument is that not all Lego pieces do actually fit together and, like, you can't cut a piece in half to make a new piece that just doesn't exist. You're limited to what comes out of the factory.
On the other hand, much to my childhood self's chagrin, even when opening a fresh set and not mixing it with other Legos, there are still quite a large number of other ways to put the pieces to together. After multiple decades my mother still sometimes tells an anecdote about how flabbergasted I was at a friend in kindergarten who opened a set I got him as a gift, ignored the instructions, and proceeded to build some bespoke edifice with virtually no resemblance to the picture on the box.
The sheer number of combinations exposed by the pieces that can number in the thousands for some sets that can be assembled either as the designer intended, completely differently, or even mixed and matched with other pieces from other sets to me feels far more like the parent comment description of exposing the underlying features, shipping a specific vision of them being assembled in a certain way, but allowing alternative visions to be constructed. What you're describing to me is more akin to building blocks; they're larger, more uniform, and only possible to put together in the ways that the designers intended. Stacking them up requires far less effort than putting together a Lego set, and you're not going to have trouble understanding how to stack some pieces so that they're relatively stable, but you're limited by the combinations of ways you can compose uniform cubes. You can't build an arch or bridge because gravity will get in the way, and you don't have the ability to make any other shapes out of the block material.
> At some point I realized I was in the opposite camp and nothing I have seen since that has really changed my view point.
I'm in this camp as well. The additional machinery required to make library features pluggable often adds a lot of complexity to the user-visible semantics of the language. And it's often not so easy to limit that complexity to only the situations where the user is trying to extend the language.
That's not always true. Sometimes the process of making seemingly core features user-replaceable will reveal simpler, more fundamental abstractions. But this article is a good example of the opposite.
One might push that analogy past its limit, and suggest that you probably wouldn't try to build a road-legal car out of Lego :).
More seriously, I'm not expecting that most people will want to use the underlying language features, nor indeed that those who do should actually do so commonly. They're there to provide for a clean separation between control logic and business logic. And that helps us to create cleaner abstractions so we can test our control logic independently of our business logic.
Most of the time I should be using the control logic we've already written, not writing more of it.
What you are missing is the whole point of writing better languages is to write better libraries.
It is not worth it to write better languages if just the "last mile" of final program is more productive --- making new languages is extraordinarily expensive, and this sort of single-shot productivity boost just isn't worth it.
If you can make better libraries --- code reuse and abstractions at hitherto unimaginable ways and then use those libraries to write yet more libraries, you get a tower of abstractions, each yielding every more productivity, like an N stage rocket.
This sort of ever-deepening library ecosystem is scarcely imaginable to most programmers. Or, they think they can imagine it, but everything looks like a leftpad waste of time to them.
Having a separation between the "pure language" and the library is a requirement if you want to have a language that can be used for low-level components, like kernels or bare-bones software.
I don't think this is possible in a language that needs a runtime, like Go.
For those trying to grok their stuff after reading the blog post, consider this.
The borrow checker vs type checker distinction is a hack, a hack that works by relegating a bunch of stuff to be "second class". Second class means that the stuff only occurs within functions, and never across function boundaries.
Proper type theories don't have this "within function, between function" distinction. Just as in the lambda calculus, you can slap a lambda around any term, in "platonic rust" you should be able to get any fragment and make it a reusable abstraction.
The author's here lens is async, which is a good point that since we need to be able to slice apart functions into smaller fragments with the boundaries at await, we need this abstraction ability. With today's Rust in contrast, the only way to do safe manual non-cheating awake would instead to be drasticly limit where one could "await" in practice, to never catch this interesting stuff in action.
In my thing I hadn't considered async at all, but was considering a kind of dual thing. Since these inconsievable types do in fact exist (in a Rust Done Right), and since we can also combine our little functions into a bigger function, then the inescable conclusion is that locations do not have a single fixed type, but have types that vary at different points in the control flow graph. (You can try model the control flow graph as a bunch of small functions and moves, but this runs afowl of non-movable stuff, including borrowed stuff, the ur-non-moveable stuff).
Finally, if we're again trying to make everything first class to have a language without cheating and frustration artificial limits on where abstraction boundaries go, we have to consider not just static locations changing type, but also pointers changing type. (We don't want to liberate some types of locations but not others.) That's where my thing comes in — references that have one type for the pointee at the beginning of the lifetime, and another type at the end.
This stuff might be mind blowing, but if should be seriously pressude. Having second class concepts in the language breeds epiccycles over time. It's how you get C++. Taking the time to make everything first class like this might be scary, but it yields a much more "stable design" that is much more likely to stand the test of time.
The post concludes by saying it's hopeless to get this stuff implemented because back compat, but I do think that that is true. (It might be hopeless for other reasons. It certainly felt hopeless in 2015.)
All this is about adding things to the language. That's backwards compatible. E.g. Drop doesn't need to be changed, because from every Drop instance a DropVer2 instance can be written instead. async v1 can also continue to exist, just by continuing to generate it's existing shitty unsafe code. And if someone wants something better, they can just use async v2 instead.
People get all freaked out about changing languages, but IMO the FUD is entirely due to sloppy imperative monkey brain. Languages are ideas, and ideas are immutable. The actual question is always, can we do "safe FFI" between two languages. Safe FFI between Rust Edition 20WX and 20YZ is so trivial that people forget to think about it that way. C and C++ is better since C "continues to exist", but of course the bar for "safe FFI" is so low when the language themselves are unsafe within themselves so that safety between them couldn't mean very much.
With harder edition breaks like this, the "safe FFI" mentality actually yields fruit.
I've been struggling with a related problem over at [1].
Feel free to read this, but it's nowhere near finished.
I'm trying to figure out how to do back references cleanly and safely.
The basic approach I'm taking is
- We can do just about everything useful with Rc, Weak, RefCell, borrow(), borrow_mut(), upgrade, and downgrade. But it's really wordy and there's a lot of run time overhead. Can we fix the ergonomics, for at least the single-owner case?
Probably. The general idea is to be able to write a field access to a weak link as
sometype.name
when what's happening under the hood is
sometype.upgrade().unwrap().borrow().name
- After fixing the ergonomics, can we fix the performance by hoisting some of the checking? Probably. It's possible to check at the drop of sometype whether anybody is using it, strongly or weakly. That allows removing some of the per-reference checking. With compiler support, we can do even more.
What I've discovered so far is that the way to write about this is to come up with real-word use cases, then work on the machinery. Otherwise you get lost in type theory.
The "Why" has to precede the "How" to get buy-in.
> The general idea is to be able to write a field access to a weak link as
sometype.name
> when what's happening under the hood is
sometype.upgrade().unwrap().borrow().name
You could easily implement this with no language-level changes as an auto-fixable compiler diagnostic. The compiler would error out when it sees the type-mismatched .name, but it would give you an easy way of changing it to its proper form. You just avoid making the .name form permanent syntactic sugar (which is way too opaque for a low-level language like Rust), it gets replaced in development.
Right. The whole problem with all this is ergonomics, from the point of view of programmers who don't want to obsess over ownership and type theory. We sort of know how to make this work. It works fine with enough Rc/Weak/etc. But it's a huge pain.
I appreciate people arguing over this. It helps. We've seen proposals from people who are too much into the type theory and not enough into ease of use. I used to do proof of correctness work, where the problem is that proof of correctness people are too into the formalism and not enough into killing bugs.
.borrow() already has a hidden unwrap. There's try-borrow(), but the assumption for .borrow() is that it will always succeed.
What I'd like to do is move as much of the checking as possible to the drop() of the owning object, and possibly to compile time. If .borrow() calls are very local, it's not too hard to determine that the lifetimes of the borrowed objects don't overlap.
Upgrade is easy to check cheaply at run time for Rc-type cells. Upgrade only fails if the owning object has been dropped. At drop, if weak_count == 0, no dangling weak references outlive the object. If there are more strong references, drop would not be called. With that check, .upgrade() will never fail.
After all, when a programmer codes a potentially fatal .borrow(), they presumably have some reason to be confident the panic won't trigger.
`.borrow()` is also a function call and thus it's not totally unexpected that it may panic.
What you're propositing however is to have the *`.name` syntax* do a lot of work and even potentially panic. That's the controversial part.
> If .borrow() calls are very local, it's not too hard to determine that the lifetimes of the borrowed objects don't overlap.
It's not too hard when you see the `.borrow()` and `.borrow_mut()`, but it becomes much harder when they become invisible and you have to check all `.field` accesses for this.
> Upgrade only fails if the owning object has been dropped. At drop, if weak_count == 0, no dangling weak references outlive the object. If there are more strong references, drop would not be called. With that check, .upgrade() will never fail.
I'm not sure what's your point here. `.upgrade()` will fail if all the owning `Rc`s have been dropped, and that's enough for this to be problematic.
> `.upgrade()` will fail if all the owning `Rc`s have been dropped, and that's enough for this to be problematic.
Want to catch that where the Rc is dropped, not at .upgrade time(). And maybe at compile time eventually.
> It's not too hard when you see the `.borrow()` and `.borrow_mut()`, but it becomes much harder when they become invisible and you have to check all `.field` accesses for this.
Good point. This is checkable at run time, but the error messages will be terrible,
because you don't know who's got the conflicting borrow. You only know it exists.
It's very similar to finding double-lock mutex deadlocks, though. Post-mortem analyzers can help. Maybe in debug mode, save enough into to produce messages such as "PANIC - double borrow at foo.rs, line 121. Conflicting borrow of the same item was at bar.rs, line 212." That's useful for locks, too. Problems of this class are all of the form "this thing here conflicts with that thing way over there", and users need to know both ends of the conflict. The Rust borrow checker is good about that.
This works much better if we have borrow lifetime checking at compile time. I've written a bit about that, and need to overhaul what I've written.
> Want to catch that where the Rc is dropped, not at .upgrade time(). And maybe at compile time eventually.
Catching that at compile time will require something that looks more like a reference into the `Rc` than a `Weak`. Most likely not the ergonomics that you want.
Catching at runtime is IMO not much different than panicking on the field access and IMO it's not enough.
> This is checkable at run time
Not sure how you check that a panic cannot occur at runtime without panicking in the first place...
> It's very similar to finding double-lock mutex deadlocks, though.
I'm not sure how they are similar. Last time I checked mutexes still required an explicit `.lock()` to be locked.
> Problems of this class are all of the form "this thing here conflicts with that thing way over there", and users need to know both ends of the conflict. The Rust borrow checker is good about that.
The last sentence doesn't make sense. You wanted to use `Rc`, `Weak` and `RefCell` precisely to avoid the kind of restrictions that are needed for the borrow checker to perform its analysis. Either you keep those restrictions and you have reference semantics or you discard them and are forced to perform runtime checks. There's no free lunch.
Your arguments basically boil down to hypothetical compile time check that doesn't exist or to arguing that hidden runtime checks are fine because you can test the code in debug mode to hopefully catch them failing. Neither of them are convincing enough.
I've considered rust for some performance-critical greenfield work where I would normally use C. Rust's syntax, idioms, and packaging are foreign to my team, so the only motivation to take that on is the safety of the borrow checker.
But as I investigate Rust, I learn of trivial use cases that cannot be safely represented [0] in Rust's syntax. TFA demonstrates even more provably-safe techniques that are impossible to express safely in Rust. So after all the difficulty of learning Rust, I might still have to choose between performance and safety?
My impression is that Rust 1.91.0 simply isn't a suitable C replacement for many real world use cases. But since it's already being used in production, I worry that backwards compatibility concerns will prevent these issues from being fixed properly, or at all.
Perhaps rust2 will get this right. Until then there's C.
I think they should just implement position-independent borrows. So instead of the borrow being an absolute pointer that gets broken if you move the self-borrowing struct, you can move it just fine.
Yes it would add like one extra add to every access, but you hardly ever need self-borrows so I think it's probably an acceptable cost in most cases.
struct A {
raw_data: Vec<u8>,
parsed_data: B<&pie raw_data>,
parsed_data2: B<&pie raw_data>
}
struct B<T> {
foo: &pie T [u8],
}
Ignoring that my made up notation doesn't make much sense, is the idea that B.foo would be an offset relative to its own adress?
So B.method(&self) might do addr(&self.foo) + self.foo, which is stable even if the parent struct A and its raw data field moves?
Then I wonder how to handle the case where the relative &pie reference itself moves. Maybe parsed_data is std::mem::replaced with parsed_data2 (or maybe one of them is an Option<B> and we Option.take() it somewhere else.)
This has been proposed at the time, but it doesn't work for the case where the borrow points to stable memory (e.g. a `&str` pointing to the contents of a `String` in the same struct). In general case a reference might point to either stable or unstable memory at runtime, so there's no way to make this always work (e.g. in async functions)
Some of the things that makes Rust hard to wrap you head around are the very things this post elucidates.
The principles exposed by the language, like lifetimes and mutability, are easy enough to understand. Yet despite understanding them, you get into epic battles with the borrow checker where the new shiny rules you've just learnt turn out to be little help in figuring out why the compiler accepts some code but rejects something similar.
You can't figure it out because these inconceivable types are not written down or explained anywhere. Instead if you are lucky you get examples of what works and what doesn't along with a hand wavy description of why, but usually it's just you versus the compiler error message. Since you don't have a concrete description of what is going to help you reason about what it likely to be accepted, you resort to experimentation to try and find what the compiler does accept. The experimental route takes ages.
If you survive enough bruising rounds of this, you will build up an empirical list of ways of working in Rust. If not, you give up on the language. To be fair, I don't think it takes a huge amount of time in the scheme or things, certainly less time than it takes to know your way around a languages standard library as that can take a month or two. However for those of us with a few languages under our belts, the time it does take comes as a shock. I'm used to it taking a few hours to become acquainted with the syntax and semantics or a new language, not finding myself still having battles with the compiler weeks later.
IMO, if the Rust doco did explain the language things the way this post does, it would make the learning it much easier.
The syntax in this post is hypothetical. In common usage you’d never encounter a need to even think about these complexities, let alone a desire to do the manual work discussed in this blog post.
I'm very much a fan of the idea that language features — and especially library features — should not have privileged access to the compiler.
Rust is generally pretty good at this, unlike (say) Go: most functionality is implemented as part of the standard library, and if I want to write my own `Vec` then (for the most part) I can. Some standard library code relies on compiler features that haven't been marked stable, which is occasionally frustrating, but the nightly compiler will let me use them if I really want to (most of the time I don't). Whereas in Go, I can't implement an equivalent to a goroutine. And even iterating over a container was "special" until generics came along.
This article was a really interesting look at where all that breaks down. There's obviously a trade-off between having to maintain all the plumbing as user-visible and therefore stable vs purely magic and able to be changed so long as you don't break the side effects. I think Rust manages to drive a fairly good compromise in allowing library implementations of core functionality while not needing to stabilise everything before releasing anything.
> I'm very much a fan of the idea that language features — and especially library features — should not have privileged access to the compiler.
At some point I realized I was in the opposite camp and nothing I have seen since that has really changed my view point.
Languages: compilers, libraries, toolkits, etc, aren't supposed to be some abstract collection of parts that can be theoretically hooked together in any possible way to achieve any possible result, they're for solving problems.
You can argue that these things are not opposites, and in theory that's true, but in practice, they seem to be! Go is a good example of making compromises that limit flexibility for the sake of developer/designer convenience.
An interesting example would be Lego, I'd argue that Go is closer to Lego design because it has a bunch of specific pieces that fit together but only in the way the designer intended.
I suspect someone taking the opposite approach from, say, Rust, would argue that some Go pieces don't actually fit together in the way that we think all lego pieces should.
My counter argument is that not all Lego pieces do actually fit together and, like, you can't cut a piece in half to make a new piece that just doesn't exist. You're limited to what comes out of the factory.
On the other hand, much to my childhood self's chagrin, even when opening a fresh set and not mixing it with other Legos, there are still quite a large number of other ways to put the pieces to together. After multiple decades my mother still sometimes tells an anecdote about how flabbergasted I was at a friend in kindergarten who opened a set I got him as a gift, ignored the instructions, and proceeded to build some bespoke edifice with virtually no resemblance to the picture on the box.
The sheer number of combinations exposed by the pieces that can number in the thousands for some sets that can be assembled either as the designer intended, completely differently, or even mixed and matched with other pieces from other sets to me feels far more like the parent comment description of exposing the underlying features, shipping a specific vision of them being assembled in a certain way, but allowing alternative visions to be constructed. What you're describing to me is more akin to building blocks; they're larger, more uniform, and only possible to put together in the ways that the designers intended. Stacking them up requires far less effort than putting together a Lego set, and you're not going to have trouble understanding how to stack some pieces so that they're relatively stable, but you're limited by the combinations of ways you can compose uniform cubes. You can't build an arch or bridge because gravity will get in the way, and you don't have the ability to make any other shapes out of the block material.
> At some point I realized I was in the opposite camp and nothing I have seen since that has really changed my view point.
I'm in this camp as well. The additional machinery required to make library features pluggable often adds a lot of complexity to the user-visible semantics of the language. And it's often not so easy to limit that complexity to only the situations where the user is trying to extend the language.
That's not always true. Sometimes the process of making seemingly core features user-replaceable will reveal simpler, more fundamental abstractions. But this article is a good example of the opposite.
One might push that analogy past its limit, and suggest that you probably wouldn't try to build a road-legal car out of Lego :).
More seriously, I'm not expecting that most people will want to use the underlying language features, nor indeed that those who do should actually do so commonly. They're there to provide for a clean separation between control logic and business logic. And that helps us to create cleaner abstractions so we can test our control logic independently of our business logic.
Most of the time I should be using the control logic we've already written, not writing more of it.
You might enjoy this video:
https://youtube.com/watch?v=D-XXZilkAXU
What you are missing is the whole point of writing better languages is to write better libraries.
It is not worth it to write better languages if just the "last mile" of final program is more productive --- making new languages is extraordinarily expensive, and this sort of single-shot productivity boost just isn't worth it.
If you can make better libraries --- code reuse and abstractions at hitherto unimaginable ways and then use those libraries to write yet more libraries, you get a tower of abstractions, each yielding every more productivity, like an N stage rocket.
This sort of ever-deepening library ecosystem is scarcely imaginable to most programmers. Or, they think they can imagine it, but everything looks like a leftpad waste of time to them.
Having a separation between the "pure language" and the library is a requirement if you want to have a language that can be used for low-level components, like kernels or bare-bones software.
I don't think this is possible in a language that needs a runtime, like Go.
> I'm very much a fan of the idea that language features — and especially library features — should not have privileged access to the compiler.
I think the reason this is that it can be less work to use compiler magic, and the result is almost (but not quite) as good.
No it is definitely more work to use compiler magic. Sometimes the perf is better, but that's it.
Oh this is really good!
I wrote https://github.com/Ericson2314/rust-papers a decade ago for a slightly different purpose, but fundamentally we agree.
For those trying to grok their stuff after reading the blog post, consider this.
The borrow checker vs type checker distinction is a hack, a hack that works by relegating a bunch of stuff to be "second class". Second class means that the stuff only occurs within functions, and never across function boundaries.
Proper type theories don't have this "within function, between function" distinction. Just as in the lambda calculus, you can slap a lambda around any term, in "platonic rust" you should be able to get any fragment and make it a reusable abstraction.
The author's here lens is async, which is a good point that since we need to be able to slice apart functions into smaller fragments with the boundaries at await, we need this abstraction ability. With today's Rust in contrast, the only way to do safe manual non-cheating awake would instead to be drasticly limit where one could "await" in practice, to never catch this interesting stuff in action.
In my thing I hadn't considered async at all, but was considering a kind of dual thing. Since these inconsievable types do in fact exist (in a Rust Done Right), and since we can also combine our little functions into a bigger function, then the inescable conclusion is that locations do not have a single fixed type, but have types that vary at different points in the control flow graph. (You can try model the control flow graph as a bunch of small functions and moves, but this runs afowl of non-movable stuff, including borrowed stuff, the ur-non-moveable stuff).
Finally, if we're again trying to make everything first class to have a language without cheating and frustration artificial limits on where abstraction boundaries go, we have to consider not just static locations changing type, but also pointers changing type. (We don't want to liberate some types of locations but not others.) That's where my thing comes in — references that have one type for the pointee at the beginning of the lifetime, and another type at the end.
This stuff might be mind blowing, but if should be seriously pressude. Having second class concepts in the language breeds epiccycles over time. It's how you get C++. Taking the time to make everything first class like this might be scary, but it yields a much more "stable design" that is much more likely to stand the test of time.
The post concludes by saying it's hopeless to get this stuff implemented because back compat, but I do think that that is true. (It might be hopeless for other reasons. It certainly felt hopeless in 2015.)
All this is about adding things to the language. That's backwards compatible. E.g. Drop doesn't need to be changed, because from every Drop instance a DropVer2 instance can be written instead. async v1 can also continue to exist, just by continuing to generate it's existing shitty unsafe code. And if someone wants something better, they can just use async v2 instead.
People get all freaked out about changing languages, but IMO the FUD is entirely due to sloppy imperative monkey brain. Languages are ideas, and ideas are immutable. The actual question is always, can we do "safe FFI" between two languages. Safe FFI between Rust Edition 20WX and 20YZ is so trivial that people forget to think about it that way. C and C++ is better since C "continues to exist", but of course the bar for "safe FFI" is so low when the language themselves are unsafe within themselves so that safety between them couldn't mean very much.
With harder edition breaks like this, the "safe FFI" mentality actually yields fruit.
This is going to take some serious reading.
I've been struggling with a related problem over at [1]. Feel free to read this, but it's nowhere near finished. I'm trying to figure out how to do back references cleanly and safely. The basic approach I'm taking is
- We can do just about everything useful with Rc, Weak, RefCell, borrow(), borrow_mut(), upgrade, and downgrade. But it's really wordy and there's a lot of run time overhead. Can we fix the ergonomics, for at least the single-owner case? Probably. The general idea is to be able to write a field access to a weak link as
when what's happening under the hood is - After fixing the ergonomics, can we fix the performance by hoisting some of the checking? Probably. It's possible to check at the drop of sometype whether anybody is using it, strongly or weakly. That allows removing some of the per-reference checking. With compiler support, we can do even more.What I've discovered so far is that the way to write about this is to come up with real-word use cases, then work on the machinery. Otherwise you get lost in type theory. The "Why" has to precede the "How" to get buy-in.
I notice this paper is (2024). Any progress?
[1] https://github.com/John-Nagle/technotes/blob/main/docs/rust/...
> The general idea is to be able to write a field access to a weak link as
> when what's happening under the hood is You could easily implement this with no language-level changes as an auto-fixable compiler diagnostic. The compiler would error out when it sees the type-mismatched .name, but it would give you an easy way of changing it to its proper form. You just avoid making the .name form permanent syntactic sugar (which is way too opaque for a low-level language like Rust), it gets replaced in development.Have you seen GhostCell[1]? Seems like this could be a solution to your problem.
[1]: https://plv.mpi-sws.org/rustbelt/ghostcell/
Yes. There's an implementation at https://github.com/matthieu-m/ghost-cell
Not clear why it never caught on.
There have been many attempts to solve the Rust back reference problem, but nothing has become popular.
The qcell crate is perhaps the most popular implementation of GhostCell-like patterns. But the ergonomics is a bit of a challenge still.
Right. The whole problem with all this is ergonomics, from the point of view of programmers who don't want to obsess over ownership and type theory. We sort of know how to make this work. It works fine with enough Rc/Weak/etc. But it's a huge pain.
I appreciate people arguing over this. It helps. We've seen proposals from people who are too much into the type theory and not enough into ease of use. I used to do proof of correctness work, where the problem is that proof of correctness people are too into the formalism and not enough into killing bugs.
> when what's happening under the hood is
> sometype.upgrade().unwrap().borrow().name
I suspect a hidden `.unwrap()` like that will be highly controversial.
.borrow() already has a hidden unwrap. There's try-borrow(), but the assumption for .borrow() is that it will always succeed.
What I'd like to do is move as much of the checking as possible to the drop() of the owning object, and possibly to compile time. If .borrow() calls are very local, it's not too hard to determine that the lifetimes of the borrowed objects don't overlap.
Upgrade is easy to check cheaply at run time for Rc-type cells. Upgrade only fails if the owning object has been dropped. At drop, if weak_count == 0, no dangling weak references outlive the object. If there are more strong references, drop would not be called. With that check, .upgrade() will never fail.
After all, when a programmer codes a potentially fatal .borrow(), they presumably have some reason to be confident the panic won't trigger.
`.borrow()` is also a function call and thus it's not totally unexpected that it may panic.
What you're propositing however is to have the *`.name` syntax* do a lot of work and even potentially panic. That's the controversial part.
> If .borrow() calls are very local, it's not too hard to determine that the lifetimes of the borrowed objects don't overlap.
It's not too hard when you see the `.borrow()` and `.borrow_mut()`, but it becomes much harder when they become invisible and you have to check all `.field` accesses for this.
> Upgrade only fails if the owning object has been dropped. At drop, if weak_count == 0, no dangling weak references outlive the object. If there are more strong references, drop would not be called. With that check, .upgrade() will never fail.
I'm not sure what's your point here. `.upgrade()` will fail if all the owning `Rc`s have been dropped, and that's enough for this to be problematic.
> `.upgrade()` will fail if all the owning `Rc`s have been dropped, and that's enough for this to be problematic.
Want to catch that where the Rc is dropped, not at .upgrade time(). And maybe at compile time eventually.
> It's not too hard when you see the `.borrow()` and `.borrow_mut()`, but it becomes much harder when they become invisible and you have to check all `.field` accesses for this.
Good point. This is checkable at run time, but the error messages will be terrible, because you don't know who's got the conflicting borrow. You only know it exists. It's very similar to finding double-lock mutex deadlocks, though. Post-mortem analyzers can help. Maybe in debug mode, save enough into to produce messages such as "PANIC - double borrow at foo.rs, line 121. Conflicting borrow of the same item was at bar.rs, line 212." That's useful for locks, too. Problems of this class are all of the form "this thing here conflicts with that thing way over there", and users need to know both ends of the conflict. The Rust borrow checker is good about that.
This works much better if we have borrow lifetime checking at compile time. I've written a bit about that, and need to overhaul what I've written.
> Want to catch that where the Rc is dropped, not at .upgrade time(). And maybe at compile time eventually.
Catching that at compile time will require something that looks more like a reference into the `Rc` than a `Weak`. Most likely not the ergonomics that you want.
Catching at runtime is IMO not much different than panicking on the field access and IMO it's not enough.
> This is checkable at run time
Not sure how you check that a panic cannot occur at runtime without panicking in the first place...
> It's very similar to finding double-lock mutex deadlocks, though.
I'm not sure how they are similar. Last time I checked mutexes still required an explicit `.lock()` to be locked.
> Problems of this class are all of the form "this thing here conflicts with that thing way over there", and users need to know both ends of the conflict. The Rust borrow checker is good about that.
The last sentence doesn't make sense. You wanted to use `Rc`, `Weak` and `RefCell` precisely to avoid the kind of restrictions that are needed for the borrow checker to perform its analysis. Either you keep those restrictions and you have reference semantics or you discard them and are forced to perform runtime checks. There's no free lunch.
Your arguments basically boil down to hypothetical compile time check that doesn't exist or to arguing that hidden runtime checks are fine because you can test the code in debug mode to hopefully catch them failing. Neither of them are convincing enough.
> But it's really wordy and there's a lot of run time overhead.
I'm curious: what do the benchmarks say about this?
I've considered rust for some performance-critical greenfield work where I would normally use C. Rust's syntax, idioms, and packaging are foreign to my team, so the only motivation to take that on is the safety of the borrow checker.
But as I investigate Rust, I learn of trivial use cases that cannot be safely represented [0] in Rust's syntax. TFA demonstrates even more provably-safe techniques that are impossible to express safely in Rust. So after all the difficulty of learning Rust, I might still have to choose between performance and safety?
My impression is that Rust 1.91.0 simply isn't a suitable C replacement for many real world use cases. But since it's already being used in production, I worry that backwards compatibility concerns will prevent these issues from being fixed properly, or at all.
Perhaps rust2 will get this right. Until then there's C.
[0]: https://databento.com/blog/why-we-didnt-rewrite-our-feed-han...
I think they should just implement position-independent borrows. So instead of the borrow being an absolute pointer that gets broken if you move the self-borrowing struct, you can move it just fine.
Yes it would add like one extra add to every access, but you hardly ever need self-borrows so I think it's probably an acceptable cost in most cases.
Say I have this type:
Ignoring that my made up notation doesn't make much sense, is the idea that B.foo would be an offset relative to its own adress?So B.method(&self) might do addr(&self.foo) + self.foo, which is stable even if the parent struct A and its raw data field moves?
Then I wonder how to handle the case where the relative &pie reference itself moves. Maybe parsed_data is std::mem::replaced with parsed_data2 (or maybe one of them is an Option<B> and we Option.take() it somewhere else.)
This has been proposed at the time, but it doesn't work for the case where the borrow points to stable memory (e.g. a `&str` pointing to the contents of a `String` in the same struct). In general case a reference might point to either stable or unstable memory at runtime, so there's no way to make this always work (e.g. in async functions)
Good point.
Rust is not an easy language.
Some of the things that makes Rust hard to wrap you head around are the very things this post elucidates.
The principles exposed by the language, like lifetimes and mutability, are easy enough to understand. Yet despite understanding them, you get into epic battles with the borrow checker where the new shiny rules you've just learnt turn out to be little help in figuring out why the compiler accepts some code but rejects something similar.
You can't figure it out because these inconceivable types are not written down or explained anywhere. Instead if you are lucky you get examples of what works and what doesn't along with a hand wavy description of why, but usually it's just you versus the compiler error message. Since you don't have a concrete description of what is going to help you reason about what it likely to be accepted, you resort to experimentation to try and find what the compiler does accept. The experimental route takes ages.
If you survive enough bruising rounds of this, you will build up an empirical list of ways of working in Rust. If not, you give up on the language. To be fair, I don't think it takes a huge amount of time in the scheme or things, certainly less time than it takes to know your way around a languages standard library as that can take a month or two. However for those of us with a few languages under our belts, the time it does take comes as a shock. I'm used to it taking a few hours to become acquainted with the syntax and semantics or a new language, not finding myself still having battles with the compiler weeks later.
IMO, if the Rust doco did explain the language things the way this post does, it would make the learning it much easier.
The syntax in this post is hypothetical. In common usage you’d never encounter a need to even think about these complexities, let alone a desire to do the manual work discussed in this blog post.
Easy is a relative measure. How familiar is a language to previous knowledge?
No, but it's the easiest language you can use on many niches.
The people who say Rust is too complex just do not want to learn. /s