From slava at dubeyko.com Wed Jun 11 03:26:58 2025 From: slava at dubeyko.com (Viacheslav Dubeyko) Date: Tue, 10 Jun 2025 17:26:58 -0700 Subject: [rulkc] [RFC] Should we consider to re-write HFS/HFS+ in Rust? Message-ID: Hello, One idea crossed my mind recently. And this is about re-writing HFS/HFS+ in Rust. It could be interesting direction but I am not sure how reasonable it could be. From one point of view, HFS/HFS+ are not critical subsystems and we can afford some experiments. From another point of view, we have enough issues in the HFS/HFS+ code and, maybe, re-working HFS/HFS+ can make the code more stable. I don't think that it's a good idea to implement the complete re- writing of the whole driver at once. However, we need a some unification and generalization of HFS/HFS+ code patterns in the form of re-usable code by both drivers. This re-usable code can be represented as by C code as by Rust code. And we can introduce this generalized code in the form of C and Rust at the same time. So, we can re-write HFS/HFS+ code gradually step by step. My point here that we could have C code and Rust code for generalized functionality of HFS/HFS+ and Kconfig would define which code will be compiled and used, finally. How do you feel about this? And can we afford such implementation efforts? Thanks, Slava. From slava at dubeyko.com Wed Jun 11 03:27:16 2025 From: slava at dubeyko.com (Viacheslav Dubeyko) Date: Tue, 10 Jun 2025 17:27:16 -0700 Subject: [rulkc] [RFC] ML infrastructure in Linux kernel Message-ID: <8ea7eba3ba0fd988cd47d79a5699d39acac729a3.camel@dubeyko.com> Hello, I would like to initiate a discussion related to an unified infrastructure for ML workloads and user-space drivers. [PROBLEM STATEMENT] Last several years have revealed two important trends: (1) moving kernel-space functionality into user-space drivers (for example, SPDK, DPDK, ublk); (2) significant number of efforts of using ML models for various real life applications (for example, tuning kernel parameters, storage device failure prediction, fail slow drive detection, and so on). Both trends represent significant importance for the evolution of the Linux kernel. From one point of view, user-space drivers represent the way of decreasing the latency and improving the performance of operations. However, from another point of view, the approach of bypassing the Linux kernel introduces security and efficiency risks, potential synchronization issues of user-space threads, and breaking the Linux kernel architecture?s paradigm. Generally speaking, direct implementation of ML approaches in Linux kernel-space is very hard, inefficient, and problematic because of practical unavailability of floating point operations in the Linux kernel, and the computational power hungry nature of ML algorithms (especially, during training phase). It is possible to state that Linux kernel needs to introduce and to unify an infrastructure as for ML approaches as for user-space drivers. [WHY DO WE NEED ML in LINUX KERNEL?] Do we really need a ML infrastructure in the Linux kernel? First of all, it is really easy to imagine a lot of down to earth applications of ML algorithms for automation of routine operations during working with Linux kernel. Moreover, potentially, the ML subsystem could be used for automated research and statistics gathering on the whole fleet of running Linux kernels. Also, the ML subsystem is capable of writing documentation, tuning kernel parameters on the fly, kernel recompilation, and even automated reporting about bugs and crashes. Generally speaking, the ML subsystem potentially can extend the Linux kernel capabilities. The main question is how? [POTENTIAL INFRASTRUCTURE VISION] Technically speaking , both cases (user-space driver and ML subsystem) require a user-space functionality that can be considered as user-space extension of Linux kernel functionality. Such approach is similar to microkernel architecture by the minimal functionality on kernel side and the main functionality on user-space side with the mandatory minimization the number of context switches between kernel-space and user-space. The key responsibility of kernel-side agent (or subsystem) is the accounting of user-space extensions, synchronization of their access to shared resources or metadata on kernel side, statistics gathering and sharing it through the sysfs or specialized log file (likewise to syslog). For example, such specialized log file(s) can be used by ML user-space extensions for executing the ML algorithms with the goal of analyzing data and available statistics. Generally speaking, the main ML logic can be executed by extension(s) on the user-space side. This ML logic can elaborate some ?recommendations?, for example, that can be shared with an ML agent on the kernel side. As a result, the kernel-space ML agent can check the shared ?recommendations? and to apply the valid ?recommendations? by means of Linux kernel tuning, recompilation, ?hot? restart and so on. Technically speaking, the user-space driver requires pretty much the same architecture as the simple kernel-space agent/subsystem and user-space extension(s). The main functionality is on the user-space side and kernel-space side delivers only accounting the user-space extensions, allocating necessary resources, synchronizing access to shared resources, and gathering statistics. Generally speaking, such an approach implies the necessity of registering a specialized driver class that could represent an ML agent or user-space driver on kernel side. Then, it will be possible to use a modprobe-like model to create an instance of ML agent or user-space driver. Finally, we will have the kernel-space agent that is connected to the user-space extension. The key point here is that the user-space extension can directly communicate with a hardware device, but the kernel-space side can account for the activity of the user-space extension and allocates resources. It is possible to suggest an unified architecture of the kernel-side agent that will be specialized by the logic of user-space extension. But the logic of the kernel-space agent should be minimal, simple, and unified as much as possible. Technically speaking, the logic of kernel-space agent can be defined by the eBPF program and eBPF arena (or shared memory between kernel-space and user- space) can be used for interaction between the kernel-space agent and the user-space extension. And such interaction could be implemented through submission and completion queues, for example. As a summary, described architecture is capable of implementing ML infrastructure in Linux kernel and unification of user-space drivers architecture. Any opinion on this? How feasible could be such vision? Thanks, Slava. From ddrokosov at salutedevices.com Wed Jun 11 12:27:37 2025 From: ddrokosov at salutedevices.com (Dmitry Rokosov) Date: Wed, 11 Jun 2025 12:27:37 +0300 Subject: [rulkc] [RFC] Should we consider to re-write HFS/HFS+ in Rust? In-Reply-To: References: Message-ID: Hello Viacheslav, Thank you for placing your ideas to RULKC mailing list. Please see my thought below. On Tue, Jun 10, 2025 at 05:26:58PM -0700, Viacheslav Dubeyko wrote: > Hello, > > One idea crossed my mind recently. And this is about re-writing > HFS/HFS+ in Rust. It could be interesting direction but I am not sure > how reasonable it could be. From one point of view, HFS/HFS+ are not > critical subsystems and we can afford some experiments. From another > point of view, we have enough issues in the HFS/HFS+ code and, maybe, > re-working HFS/HFS+ can make the code more stable. > > I don't think that it's a good idea to implement the complete re- > writing of the whole driver at once. However, we need a some > unification and generalization of HFS/HFS+ code patterns in the form of > re-usable code by both drivers. This re-usable code can be represented > as by C code as by Rust code. And we can introduce this generalized > code in the form of C and Rust at the same time. So, we can re-write > HFS/HFS+ code gradually step by step. My point here that we could have > C code and Rust code for generalized functionality of HFS/HFS+ and > Kconfig would define which code will be compiled and used, finally. > > How do you feel about this? And can we afford such implementation > efforts? I followed the development of HFS/HFS+ a long time ago, but AFAIK, this filesystem is now considered legacy from Apple's perspective. Out of curiosity, do you know of any real-world use cases for HFS/HFS+ outside of older macOS versions? Also, I found a Rust filesystem (FS) example that demonstrates possible interactions with the VFS. Would that be sufficient to completely rewrite a large filesystem like HFS/HFS+? https://github.com/Rust-for-Linux/linux/blob/rust/samples/rust/rust_fs.rs [...] -- Thank you, Dmitry From slava at dubeyko.com Thu Jun 12 00:32:24 2025 From: slava at dubeyko.com (Viacheslav Dubeyko) Date: Wed, 11 Jun 2025 14:32:24 -0700 Subject: [rulkc] [RFC] Should we consider to re-write HFS/HFS+ in Rust? In-Reply-To: References: Message-ID: <54b0b1ec543f9324c901787efda04e50df8c218d.camel@dubeyko.com> On Wed, 2025-06-11 at 12:27 +0300, Dmitry Rokosov wrote: > Hello Viacheslav, > > Thank you for placing your ideas to RULKC mailing list. > Please see my thought below. > > On Tue, Jun 10, 2025 at 05:26:58PM -0700, Viacheslav Dubeyko wrote: > > Hello, > > > > One idea crossed my mind recently. And this is about re-writing > > HFS/HFS+ in Rust. It could be interesting direction but I am not > > sure > > how reasonable it could be. From one point of view, HFS/HFS+ are > > not > > critical subsystems and we can afford some experiments. From > > another > > point of view, we have enough issues in the HFS/HFS+ code and, > > maybe, > > re-working HFS/HFS+ can make the code more stable. > > > > I don't think that it's a good idea to implement the complete re- > > writing of the whole driver at once. However, we need a some > > unification and generalization of HFS/HFS+ code patterns in the > > form of > > re-usable code by both drivers. This re-usable code can be > > represented > > as by C code as by Rust code. And we can introduce this generalized > > code in the form of C and Rust at the same time. So, we can re- > > write > > HFS/HFS+ code gradually step by step. My point here that we could > > have > > C code and Rust code for generalized functionality of HFS/HFS+ and > > Kconfig would define which code will be compiled and used, finally. > > > > How do you feel about this? And can we afford such implementation > > efforts? > > I followed the development of HFS/HFS+ a long time ago, but AFAIK, > this > filesystem is now considered legacy from Apple's perspective.? Out of > curiosity, do you know of any real-world use cases for HFS/HFS+ > outside > of older macOS versions? > Even I didn't follow to HFS/HFS+ development because the active phase was more than 20 years ago. :) The HFS/HFS+ code maintainership was dead more than 10 years already. I tried to contribute to HFS/HFS+ 10 years ago and it was really sad experience. The main use-case of HFS/HFS+ file system(s) was and is of mounting and accessing under Linux file system volumes that was created and filled by data under Mac OS X. I don't think that anybody now would like to use HFS/HFS+ file system as rootfs or regular data partition under Linux. Also, John Paul Adrian Glaubitz shared this "We definitely need this driver to stay for Debian Ports as otherwise we won't be able to boot PowerMacs using GRUB.?GRUB won't be usable anymore on PowerMacs with HFS/HFS+ being removed from the kernel.". And this is exactly the point. The HFS/HFS+ is not critical subsystems in the kernel and it can be easily re-written in Rust without significant confrontation. Technically speaking, potentially, HFS/HFS+ could be the first Linux kernel file system driver(s) re-written in Rust. I am a HFS/HFS+ maintainer in Linux kernel and I am ready to support and promote the Rust implementation of HFS/HFS+ in Linux kernel. > Also, I found a Rust filesystem (FS) example that demonstrates > possible > interactions with the VFS. Would that be sufficient to completely > rewrite a large filesystem like HFS/HFS+? > The HFS/HFS+ are tiny file system drivers compared with other Linux kernel file systems. However, I don't want to re-write the whole HFS or HFS+ driver in Rust at once. The both HFS and HFS+ drivers have a lot of shared logic that is duplicated in both drivers. So, I would like to see at first something like Rust "library" that both HFS and HFS+ drivers could use by calling shared functionality. Finally, as the second step, we could have Rust implementation of the whole driver. And, as the maintainer, I would like to find Rust guys that could be interested to implement such functionality for HFS/HFS+ because it looks like promoting new technologies in Linux kernel. And, HFS/HFS+ could benefit from memory-safety features of Rust. Thanks, Slava. > https://github.com/Rust-for-Linux/linux/blob/rust/samples/rust/rust_fs.rs > > [...]