I work in the embedded space. I'd absolutely love to have a tool to build immutable, signed distro images which I can push to devices with a/b style updates. I imagine you can do this with mkosi, but it doesn't quite feel like the intended use case.
Most immutable distros (this included) are developed with the idea that you'd run local commands on the machine, and update a single OS image in place. That gets pretty unwieldy to manage across a fleet of devices.
Right now the industry standard tooling for building truly immutable images for embedded devices is Yocto. It works well, but it's incredibly complicated and has ridiculous build times because it insists on building every single package from source. It's utter madness that building a Linux image for a common ARM64 processor requires me to compile the Rust compiler myself.
You don't have to build ParticleOS images on the machine itself, it's perfectly possible to build them offline somewhere else and download them from the target machine when doing an update with systemd-sysupdate. It's just that we haven't quite gotten to ironing out all the details there. We're adding support to OBS (OpenSUSE Build System) to build ParticleOS images and will eventually start publishing prebuilt images there which can then be consumed by systemd-sysupdate.
For the embedded space you'd just build the ParticleOS images on your own build system, publish them somewhere and consume them with systemd-sysupdate, doesn't have to be OBS of course.
But we don't do stuff like only downloading diffs with systemd-sysupdate and such yet, so your milleage may vary.
That's interesting. I've been looking for an angle on embedded Linux software updates for my deployment tool (currently limited to updating desktop and server apps only).
When you say push, do you literally mean push or do you mean the devices would still pull updates on their own schedule where you get to control rollout percentages, channels, etc centrally? Mostly devices have to be assumed to be behind NAT, right?
What I'm thinking here is maybe it'd be useful to have a mode in Conveyor [1] that builds a standalone ARM Linux that can boot and pull updates for everything including the app, but which coordinates with the userspace app to decide when it's time to restart and for which the build to update to is controlled by server-supplied metadata so you can divide the fleet by tag, etc.
The problem with immutable images is you need a complimentary config management tool to manage them at scale. That's why we built Etcha[0] (config management for 1000+ devices) and then built an immutable OS (EtchaOS) around it--they're meant to work together instead of "cloud-init is good enough". cloud-init is terrible at scale, especially outside of public cloud.
We at darkscience used buildroot for a simple ZNC SBC[0] distribution which we deployed to an Orange Pi Zero that we sold at 33C3[1].
It was great! I highly recommend!
It didn't take long to get started and while the compile times were pretty large (wonder if NIX or Bazel could help here?) it ended up with absolutely microscopic resource requirements.
I've had a reasonable amount of luck with a combination of Buildroot and Bazel. I use Buildroot to assemble a minimal Base OS with just the kernel, bootloader and a few system services. Then the application layer is built with Bazel and assembled into a set of squashfs images which are mounted as overlay filesystems at boot time. The whole thing is shipped as a SWUpdate file built with the Bazel layer.
Because most of the iteration is happening in the Bazel layers I can generate a full system update in about 15 seconds with everything being fully reproducible.
> Most immutable distros (this included) are developed with the idea that you'd run local commands on the machine, and update a single OS image in place. That gets pretty unwieldy to manage across a fleet of devices.
This is how it always has been on enterprise network devices like switches, routers, etc. It’s pretty trivial to automate updates.
I’m somewhat a fan of systemd and I’m interested in immutable images, but I don’t want to build systemd from source:
>Sometimes ParticleOS adopts systemd features as soon as they get merged into systemd without waiting for an official release. As a result it's recommended to build systemd from source when building ParticleOS to make sure all required features are supported
> I’m somewhat a fan of systemd and I’m interested in immutable images, but I don’t want to build systemd from source:
You don't have to, we build packages and publish repositories from latest main, and the particle configs can use those by enabling the 'obs' build profile
As in: "you've tried alternatives and you really think that systemd is superior", or as in "you don't see why people complain about systemd and its philosophy"?
Genuinely interested, because usually people I hear say that they don't see the problem with systemd don't really have experience with the alternatives. Which makes sense: people who like being able to use alternatives tend to get screwed by systemd :-).
Edit: for those downvoting, this is a genuine question. I mostly hear opinions from people who hate systemd, and those who don't generally just don't really care: "I've always used systemd and I haven't had issues". But the latter group wouldn't call themselves "fans of systemd".
Systemd makes me sad because 99% of what I like about it could have been implemented in a modular fashion, but instead it's mostly all-or-nothing. I still use OpenRC on servers, but I have to implement or integrate a lot of the missing features, like all the cgroup/namespace stuff, logging, etc.
This. I don't like that it's not modular, and that more and more projects simply depend on on it. It feels like it makes it harder to not have systemd.
Which makes sense for systemd if their goal is to kill the "competition". But that's not my vision of how open source should work.
> But that's not my vision of how open source should work.
Exactly. I expect a modular fork of systemd when people get annoyed enough. I still don't understand why people treat systemd as the second coming of the Christ.
Disclosure: I'm managing large number of servers for the last 20 years. Don't tell me that init scripts were bad. I won't buy it. Don't tell me systemd is fast, again, I won't buy it.
> Exactly. I expect a modular fork of systemd when people get annoyed enough.
That's exactly the case with elogind, used by e.g. Alpine, Void, and Gentoo.
> I still don't understand why people treat systemd as the second coming of the Christ.
FWIW it managed to pull most distros in a single direction, decreasing ecosystem fragmentation. The outliers are well, outliers.
Just to be clear: it's not an endorsement from my side; there's enough criticism of systemd which I will not repeat. But the fragmentation aspect is what kills so much momentum, and it just runs too deep: Gnome was started because KDE's dependency on Qt wasn't kosher; Qt later got relicensed, but both projects kept pulling in different directions. It feels like any compatibility between anything is coincidental.
> Qt later got relicensed, but both projects kept pulling in different directions.
Both desktop environments pull in different directions in appearance only. Esp. KDE team works their heads off to be compatible with GNOME and to support/build standards which work equally well on both systems. For example, ".desktop" file specification. Another notable example is, every KDE release comes with identical GTK themes for flagship Qt theme, plus numerous others which I can't remember now.
So, the underlying standards of many Desktop Environments are the same, because there are standards which many don't see, know, care (and this is a good thing). Vaxry is a case study here, and I exclude his creations.
I know this, because, I took part in developing a Debian derivative distro for ~5 years or so.
> FWIW it managed to pull most distros in a single direction, decreasing ecosystem fragmentation. The outliers are well, outliers.
On the other hand, every distribution selects parts of systemd, patches part of the selected set and configures this set, hence integrates systemd differently in different levels. This creates behavioral differences in systems and their systemd installations.
IOW, systemd didn't reduce fragmentation to the level people wanted them to believe.
Moreover, pulling from that 20+ experience with Linux systems, their behavior was already this consistent since forever. RedHat had a wrapper called "service" on top of "/etc/init.d" system, but the scripts were portable. Some config files unrelated to systemd/init.d were in different places (and still are). e.g. apache/httpd, /etc/network/interfaces vs. /etc/sysconfig/network-scripts, etc.
So, the defragmentation by systemd is an illusion. Since forever, I had to either look for /etc/{redhat,debian}_version or type apt/yum/dnf and press enter to see which distro I'm on. The rest is basically the same, across hundreds of installations.
> Both desktop environments pull in different directions in appearance only.
Nope, no. KDE has a dozen toggles for every feature, half of which don't make sense. It has OK/Cancel/Apply, as if Windows 95 was peak UX. Konsole uses the same color theme regardless of whether you pick light/dark; I've tried to hack together a script, but the CLIs/APIs don't make sense: you have to detect the light/dark preference by parsing the current system theme name. Good luck if it's something like some variation of "PurPurDay"/"PurPurNight". I also couldn't find a way to get programmatically notified about the theme changing - but KDE still shoves useless notifications in my face, like "button X was removed from the panel", which you have to disable one by one. KDE can't tell "powerful" from "overwhelming".
Gnome has a top bar. It has three buttons, the rest is wasted space. The "dock" can't be moved to the left/right screen edge - more precious vertical space taken away. It has no desktop icons, even though there's a "Desktop" folder in Nautilus. It can't set up different screen dim/lock timeouts on battery vs AC. Meanwhile I was watching a PR[1] that would add iCloud integration to system accounts - it got rejected, because "users can just set up IMAP/SMTP". Gnome can't tell "simple" from "simplistic".
I could go on, but please use macOS for a week, and try to draw a fair comparison.
> KDE team works their heads off to be compatible with GNOME [...].
Here we go. In the other thread, I was arguing that "competition" in this space is just wasted effort.
> So, the underlying standards of many Desktop Environments are the same [...].
So if I want to write a native application, which toolkit should I target? On macOS, it's Cocoa. On Windows, it's "whatever Windows is doing in this release", but at least I can still run programs that weren't touched since 1997.
On Linux, I have a choice of plain Qt vs KDE, plain GTK (2/3/4) vs Adwaita, and so on. Apps written with one look and behave out of place when used with the other. Every time glibc breaks something, the app needs to be rebuilt. OK, so I want to provide users with a self-contained package. The choices are Flatpak, Snap, and AppImage, all of which are built differently. I could go on.
> Another notable example is, every KDE release comes with identical GTK themes for flagship Qt theme [...].
Design is not how it looks, it's how it works.
> Moreover, pulling from that 20+ experience with Linux systems [...].
Yeah, I've also used Linux on the desktop for 20 years. Then I tried macOS for a month. Then tried OS X 10.5 for a comparison. I remember considering a MacBook in 2007, I regret not getting one back then.
> Since forever, I had to either look for /etc/{redhat,debian}_version or type apt/yum/dnf and press enter to see which distro I'm on.
I've inherited and managed a heterogenous fleet of Debian, Ubuntu, Fedora, CentOS, OpenBSD, etc, (many on different releases), and had to write a lot of Ansible roles by hand, because few existing roles could handle all of these. The quirks and hacks only piled and piled. It's insane.
I’ve used various forms of S6 init, you might take a look. It has its own scripting called execline, which is good, but you can also just use any shell. It doesn’t do everything systemd does but it’s quite small and very modular. https://skarnet.org/software/s6-linux-init/why.html
I like the general idea of execline, but I feel like it relies too much on chaining. It's a bit like Lisp, but without any parens - it hurts readability.
It went from "shells have too much syntax, which creates security issues" straight into "forth with blocks". My idea: runit already provides chpst, which folds things like cd, chroot, nice, etc into a single command; so let's put it all into a block with named parameters. Like this:
run {
dir: /usr/src/foo
script: {
./configure
make -j(nproc)
make install
}
nice: 20
ionice: idle
}
Maybe it could draw a bit from rc, which has actual lists as a first-class data structure.
I'm not exactly sure about the syntax, but that's the general direction.
I've used Solaris SMF back in the day, which is what systemd largely copied, but badly.
The major difference is, in SMF you specify services, their dependencies and whatever else you need. Then SMF looks at the whole desired system, computes the dependency tree and boot order, validates everything and writes a boot plan that can be executed in a static and deterministic fashion. You can look at the plan, execute that exact same plan each boot. You therefore know that if the system came up once using that plan it will again.
Whereas systemd does everything on the fly, the boot order is computed anew with each boot, all the services on the same tree level are started at the same time and just have to fight it out. Thus the actual boot order varies greatly with each boot and will fail randomly, wait randomly (hands up: who wasn't annoyed by the systemd-rememberance-minute-and-a-half popping up like an inopportune windows update?) and will be hard to debug because you just cannot reproduce anything that lead to that. Your boot will be littered with 'wait-for-something' units, 'sleep-for-10s-because-that-magically-makes-it-work' and additional dependencies to maybe clean up the tree which the maintainers forgot about. Sometimes intentionally because "systemd is about making it boot fast, and it works on my laptop".
It isn't all bad, but for systems that I need to boot and shutdown reliably, systemd is a step back even over init scripts because of the nondeterminism. You can "fix" things by just watchdog-rebooting when the system doesn't come up, and the downtime will just be a little longer. But it is annoying. And it is a design bug that will never be fixed.
Systemd has more in common with launchd than SMF to be fair. SMF very much is a product of another time. Completely fine when we were dealing with workstations whose configuration was mostly static and not expected to change but very much unfit to the environment systemd operates nowadays.
Systemd uses a dependency graph by the way. There is little magic. Provided you have properly setup what depends on what, everything will boot just fine and it’s somehow resilient enough than even if you messed up somewhere there is a good chance it manages to bring everything up after some hiccups.
OTOH figuring things out at runtime seems to be how it paints itself into corners and just... stops. I'm not going to pretend that sysv was good, but when a sysv box booted, it came up, and when I told it to shut down, it died on the spot; there was none of this sitting staring at it twiddling its thumbs. On the bright side, these days things do eventually time out; I don't miss CentOS 7 freezing forever on shutdown.
Regarding "there is little magic", Poettering repeatedly praises socket activation in his blog. It's not a new idea, but an old one that fell out of favor because it kinda sucked (remember xinetd?). It's about pretending that a service is running when by providing its listening socket, then starting it when the first client connects.
It has some problems: it's more complicated, it can actually slow things down compared to starting a service as soon as possible, and it has new error cases because of course "pretending that a service is running" doesn't have exactly the same effect as actually having the service running.
Socket activation is entirely optional and is not what I would personally call "magic".
I don't think "pretending that a service is running" is a fair description of what it does however. It's just buffering. It has advantages in some case which is why it exists. It doesn't "kinda suck".
Entirely optional, but widely used. An advertised advantage is that service A that relies on B can start before or concurrently with B. Calling that "just buffering" is really not accurate.
You are not crazy. The fact that we managed to make a livable space in and pull electricity and utilities into the shanty-town-systemd we all live in nowadays is not proof of great design.
From my own experience the alternatives are fine as long as the distro maintainers put on the effort to make them work.
I've used Devuan and others with runit, sysV, s6 and openrc. And honestly, openrc is the better alternative from an user perspective.
Nevertheless, it is hard to argue against systemd when the critical mass is there, and you run into eventual problems.
Still, I find alternatives such as runit and openrc easier to reason about than systemd. Perhaps this is due to the need to fully grok them, so you're able to fix issues yourself. Contrast that with systemd, where there are plenty of drag and drop solutions out there, so that requirement is lessened. You could also argue the real reason is systemd's complexity, but personally that doesn't feel like the core reason.
Either way, it's good that there are alternatives. I like how heterogenous the whole Linux ecosystem can be, it helps mitigate large scale security issues.
> Either way, it's good that there are alternatives. I like how heterogenous the whole Linux ecosystem can be, it helps mitigate large scale security issues.
IMHO all it does is drive fragmentation and marginalisation. Either your alternative is successful enough that third parties are compelled to care about it (making the integration work on their part more painful, which is often "solved" by using Electron); or it isn't, so you have to bring in your own integration (aka glue).
Sorry but Ballmer was right: developers, developers, developers. Users add value, developers multiply it.
But I don't see it as fragmentation, but as competition. The better idea tends to prevail, and it's nice that the environment allowed alternatives to even try :)
hell, systemd itself happened as a esult of this environment :)
Some competition is OK. For example, similar programming languages (e.g. Rust and Zig) - they inspire each other, fit slightly different niches, and there's plenty of space for both.
Experimentation is OK. DNF started as a yum fork, with a different solver algorithm. IMHO yum should've just adopted the algorithm, but Fedora/CentOS ended up switching. That's mostly fine (people had to update docs, retrain their muscle memory).
The level of "competition" we see in package managers, desktop environments, initrd builders (!), resolv.conf managers (!!!), is just wasted effort. There's no "pick the winner", it's "everyone is a loser".
DNF didn't start as a competitive fork of Yum, it started as the Python 3 rewrite of Yum from the same developers. It was really just Yum version 4, using a different name so it could be installed in parallel with Yum for an extended period of time. Yum changing algorithms was never on the table, as that Python 2 code had to get phased out. Fedora, CentOS, and RHEL switching to it was always the plan.
We made heavy use of systemd and its related components on a fleet of a few thousand edge devices and it worked great for us. Occasionally, we would run into bugs or confusing behaviour and have to dig through GitHub issues, but that was far outweighed by the problems it took care of for us.
systemd does a lot more than just init. You can preach about whatever alternative inits all you like, but they don't do even 10% of what systemd does.
> systemd does a lot more than just init. You can preach about whatever alternative inits all you like, but they don't do even 10% of what systemd does.
I didn't mean to preach about alternative inits.
My personal criticism of systemd is that it tries to take over everything, limiting the choice.
I want to have a choice, that's all. And systemd doesn't do anything to help there.
Modularity would be to be able to use, say, systemd-logind or systemd-udev without using systemd as your init, as there is no tangible or important dependency on the use of systemd as init for those functions.
On a fully-systemd system you can opt out of systemd features. But you cannot opt in to systemd features on a non-systemd system.
> there is no tangible or important dependency on the use of systemd as init for those functions.
How do you catch device events on very early startup (e.g., between start of pid1 and when processes start to fire?) if the thing that manages the devices isn't integrated with pid1 at a deep level?
logind does not "manage devices", it just provides an API for your display server to open privileged I/O devices and get a file descriptor, and have these file descriptors revoked later.
The difference with a plain `open` is that logind will allow only one process at a time to have access, and only for sessions that logged in physically (tracked using PAM), and does not require the process to have direct file access (which could enable key-logging). It will also revoke the file descriptors when switching sessions.
logind's other random features like dock and power button triggers can be done whenever logind is running and ready - the underlying API reports levels and are not timing critical.
The only thing logind really uses at runtime from systemd is temporary storage of file descriptors that it can get back after restart for graceful upgrades.
Projects like elogind and eudev exists that painstakingly strip out the systemd portions while retaining the functionality, but it's an uphill battle to constantly redo every time there's a change.
I am telling you that it is difficult to use systemd subcomponents without running all of systemd, and you give me example of running most of systemd and replacing a few subcomponents.
>I am telling you that it is difficult to use systemd subcomponents without running all of systemd
Actually it's the first time you have stated that, but I'll bite.
How do you propose that would work, and why would you even want to do that? The entire point of having these modules is for them to plug in to core systemd, which enables a lot of magic that would otherwise require an amalgam of shitty shell scripts. If you don't want to use systemd, why not just use chronyd for NTP, dnsmasq for DNS, and so on? Your problem is completely imaginary.
> Actually it's the first time you have stated that, but I'll bite.
Fair point, I said it in another comment :-).
> How do you propose that would work
I don't. I merely state the fact that I don't like the way it's going with systemd, which is that many projects now assume that you have systemd and depend on it. And if you don't have systemd, you're screwed.
Similar to projects that assume that you have Ubuntu and depend on Ubuntu specificities.
And again I'm not saying that it should be illegal; just that I don't like it.
People who don't like it can call me back when they graduate beyond ricing their i3 desktop and run a ~5k node server fleet, and then tell me how they did it without reimplementing everything systemd already does.
That is why RedHat wants it. What does it help on the laptop that I am writing this?
And what happens when you upgrade that fleet (unless you're doing clean installs) and sytemd-resolved is not resolving anything? (Just the most recent thing I've had)
Well the key is to always do clean-installs and join the fleet and decommission the old machines. Machines are essentially immutable and disposable, so upgrades are done with full machines. And if anything hardware goes bad on whole or sometimes half racks.
Not the person you're asking, but I'm curious what you mean when you say "the alternatives".
I've used runit, openrc, briefly upstart (what a trashfire that was), and I don't miss a single thing about them while writing systemd unit files.
I've used GNU Shepherd on Guix, and honestly, meh, systemd feels way more pragmatic.
I've also used systemd-networkd as well as NetworkManager, and feel like they both still have a few warts, but overall prefer systemd-networked on servers.
I've used grub, refit, and systemd-boot, and gotta say systemd-boot is my preference there too.
I haven't really used systemd portable services, but I've used docker and don't like it, and my overall impression is I'll probably like portable services more.
Are there other alternatives I have to use before I'm allowed to have an opinion? What alternatives have you used which made you feel so enlightened?
Personally, it's not that the alternatives are so awesome.
It's when i meet against some awful design choices in systemd, like how it decides to wait forever rather than hard fail, and by wait forever, i mean wait for 30 seconds, no, 1 minute 30 seconds, no 5 minutes... you get it.
Or you can easily lock yourself out when there's a typo in /etc/fstab.
God forbid you used cgroupsv1 for anything when poettering unilaterally decided to punish everyone with a 30 second delay for using it.
If you stray off of the intended opinionated path, be prepared for the least user-friendly experience linux has to offer. You might even get some abuse from the developers if you try to ask for help, but they've probably toned it down since CoC became a thing (I hope).
God forbid you used cgroupsv1 for anything when poettering unilaterally decided to punish everyone with a 30 second delay for using it.
Intentional time-delays are a nice middle ground. Yes, it is annoying to the users who've ignored all prior deprecation warnings, but it's better than those same users being "surprised" when support is removed entirely. Maybe you'd consider it less of a "punishment" if systemd dropped support for cgroupsv1 outright instead of inserting a time delay, I believe the opposite.
> What alternatives have you used which made you feel so enlightened?
I didn't mean to seem like I feel so enlightened.
My experience is that when not using systemd, I regularly get screwed by what I would call the "imperialist culture" of systemd, for lack of better words: it doesn't feel like you can relatively easily extract subcomponents of systemd and use it standalone. And more and more projects depend on a particular subcomponent of systemd, meaning that they require systemd.
So it's not so much that I believe systemd is technically worse than the alternatives. It's more the philosophy that I don't like.
Blatantly not the person you're asking, but once you go past basic service management trying to do it just about anywhere else is an absolute and total nightmare. People who have major problems with systemd more than anything just ever seem confused, often vastly overestimating systemd's actual scope.
The reality to many is its the simplest simple way to handle some very complex tasks, and the vast majority of it is still all so very very much optional even in the most 'systemd heavy' distros.
Is ParticleOS also a useful everyday driver for Linux enthusiasts who aren't systemd developers? Let's say I got bored with Gentoo and I want to try something new. Is ParticleOS something that a technical user would enjoy at the current stage?
I think you'll get the most out of this at the moment if you're interested in actively contributing to systemd. There's lots of work left to be done to make this usable which will just be annoying if you aren't interested in fixing some of this stuff yourself.
Of course if you're interested in doing small prs and such to improve things then I think it could be very satisfying. You can always join the mkosi matrix channel to chat more about this stuff.
I'm not sure nix is aimed at the same kind of things here. ParticleOS seems to be a logical continuation of the effort that systemd has been putting into TPM2, boot security, signing, and immutability.
> Currently both arch and fedora are supported distributions.
Does this mean that we can now have an actual immutable Arch-based distro?!
I love Silverblue, best distro ever, but the day I can have immutable Arch Linux I’m outta here! I truly despise rpm and all the fedora machinations required to build a custom package.
Yeah I've been running the Arch variant of this on my personal laptop for a while now. Of course no stability guarantees with this for now since you'll be running systemd from source.
mkosi runs pacman for you and then packs up the result as a disk image. It also builds a unified kernel image and does a bunch of signing. The new /usr partition and UKI are then installed with systemd-sysupdate.
Building the next update locally is slow because erofs has to compress the entirety of /usr on my rather old laptop and that takes a while.
Aside from that, I'm using flatpak for firefox and for some reason it takes absolutely forever (like > 15s) to start firefox on my machine, still need to look into why that happens.
Aside from those it's very usable. But of course it's running systemd from source so I'm not going to actually recommend anyone to run this who is not a systemd maintainer until we iron out the kinks.
I work in the embedded space. I'd absolutely love to have a tool to build immutable, signed distro images which I can push to devices with a/b style updates. I imagine you can do this with mkosi, but it doesn't quite feel like the intended use case.
Most immutable distros (this included) are developed with the idea that you'd run local commands on the machine, and update a single OS image in place. That gets pretty unwieldy to manage across a fleet of devices.
Right now the industry standard tooling for building truly immutable images for embedded devices is Yocto. It works well, but it's incredibly complicated and has ridiculous build times because it insists on building every single package from source. It's utter madness that building a Linux image for a common ARM64 processor requires me to compile the Rust compiler myself.
You don't have to build ParticleOS images on the machine itself, it's perfectly possible to build them offline somewhere else and download them from the target machine when doing an update with systemd-sysupdate. It's just that we haven't quite gotten to ironing out all the details there. We're adding support to OBS (OpenSUSE Build System) to build ParticleOS images and will eventually start publishing prebuilt images there which can then be consumed by systemd-sysupdate.
For the embedded space you'd just build the ParticleOS images on your own build system, publish them somewhere and consume them with systemd-sysupdate, doesn't have to be OBS of course.
But we don't do stuff like only downloading diffs with systemd-sysupdate and such yet, so your milleage may vary.
That's interesting. I've been looking for an angle on embedded Linux software updates for my deployment tool (currently limited to updating desktop and server apps only).
When you say push, do you literally mean push or do you mean the devices would still pull updates on their own schedule where you get to control rollout percentages, channels, etc centrally? Mostly devices have to be assumed to be behind NAT, right?
What I'm thinking here is maybe it'd be useful to have a mode in Conveyor [1] that builds a standalone ARM Linux that can boot and pull updates for everything including the app, but which coordinates with the userspace app to decide when it's time to restart and for which the build to update to is controlled by server-supplied metadata so you can divide the fleet by tag, etc.
[1] https://hydraulic.dev/
The problem with immutable images is you need a complimentary config management tool to manage them at scale. That's why we built Etcha[0] (config management for 1000+ devices) and then built an immutable OS (EtchaOS) around it--they're meant to work together instead of "cloud-init is good enough". cloud-init is terrible at scale, especially outside of public cloud.
0 - https://etcha.dev
If only Tiny Core Linux had had better tooling and if only corporations liked simplicity, it might have been more popular…
Bulidroot is popular and simple alternative. But corporations prefer complexity.
We at darkscience used buildroot for a simple ZNC SBC[0] distribution which we deployed to an Orange Pi Zero that we sold at 33C3[1].
It was great! I highly recommend!
It didn't take long to get started and while the compile times were pretty large (wonder if NIX or Bazel could help here?) it ended up with absolutely microscopic resource requirements.
[0]: https://git.drk.sc/Derecho/irc-sbc-buildroot
[1]: https://media.ccc.de/c/33c3
I've had a reasonable amount of luck with a combination of Buildroot and Bazel. I use Buildroot to assemble a minimal Base OS with just the kernel, bootloader and a few system services. Then the application layer is built with Bazel and assembled into a set of squashfs images which are mounted as overlay filesystems at boot time. The whole thing is shipped as a SWUpdate file built with the Bazel layer.
Because most of the iteration is happening in the Bazel layers I can generate a full system update in about 15 seconds with everything being fully reproducible.
> Most immutable distros (this included) are developed with the idea that you'd run local commands on the machine, and update a single OS image in place. That gets pretty unwieldy to manage across a fleet of devices.
This is how it always has been on enterprise network devices like switches, routers, etc. It’s pretty trivial to automate updates.
I’m somewhat a fan of systemd and I’m interested in immutable images, but I don’t want to build systemd from source:
>Sometimes ParticleOS adopts systemd features as soon as they get merged into systemd without waiting for an official release. As a result it's recommended to build systemd from source when building ParticleOS to make sure all required features are supported
> I’m somewhat a fan of systemd and I’m interested in immutable images, but I don’t want to build systemd from source:
You don't have to, we build packages and publish repositories from latest main, and the particle configs can use those by enabling the 'obs' build profile
> I’m somewhat a fan of systemd
As in: "you've tried alternatives and you really think that systemd is superior", or as in "you don't see why people complain about systemd and its philosophy"?
Genuinely interested, because usually people I hear say that they don't see the problem with systemd don't really have experience with the alternatives. Which makes sense: people who like being able to use alternatives tend to get screwed by systemd :-).
Edit: for those downvoting, this is a genuine question. I mostly hear opinions from people who hate systemd, and those who don't generally just don't really care: "I've always used systemd and I haven't had issues". But the latter group wouldn't call themselves "fans of systemd".
Systemd makes me sad because 99% of what I like about it could have been implemented in a modular fashion, but instead it's mostly all-or-nothing. I still use OpenRC on servers, but I have to implement or integrate a lot of the missing features, like all the cgroup/namespace stuff, logging, etc.
This. I don't like that it's not modular, and that more and more projects simply depend on on it. It feels like it makes it harder to not have systemd.
Which makes sense for systemd if their goal is to kill the "competition". But that's not my vision of how open source should work.
> But that's not my vision of how open source should work.
Exactly. I expect a modular fork of systemd when people get annoyed enough. I still don't understand why people treat systemd as the second coming of the Christ.
Disclosure: I'm managing large number of servers for the last 20 years. Don't tell me that init scripts were bad. I won't buy it. Don't tell me systemd is fast, again, I won't buy it.
> Exactly. I expect a modular fork of systemd when people get annoyed enough.
That's exactly the case with elogind, used by e.g. Alpine, Void, and Gentoo.
> I still don't understand why people treat systemd as the second coming of the Christ.
FWIW it managed to pull most distros in a single direction, decreasing ecosystem fragmentation. The outliers are well, outliers.
Just to be clear: it's not an endorsement from my side; there's enough criticism of systemd which I will not repeat. But the fragmentation aspect is what kills so much momentum, and it just runs too deep: Gnome was started because KDE's dependency on Qt wasn't kosher; Qt later got relicensed, but both projects kept pulling in different directions. It feels like any compatibility between anything is coincidental.
> Qt later got relicensed, but both projects kept pulling in different directions.
Both desktop environments pull in different directions in appearance only. Esp. KDE team works their heads off to be compatible with GNOME and to support/build standards which work equally well on both systems. For example, ".desktop" file specification. Another notable example is, every KDE release comes with identical GTK themes for flagship Qt theme, plus numerous others which I can't remember now.
So, the underlying standards of many Desktop Environments are the same, because there are standards which many don't see, know, care (and this is a good thing). Vaxry is a case study here, and I exclude his creations.
I know this, because, I took part in developing a Debian derivative distro for ~5 years or so.
> FWIW it managed to pull most distros in a single direction, decreasing ecosystem fragmentation. The outliers are well, outliers.
On the other hand, every distribution selects parts of systemd, patches part of the selected set and configures this set, hence integrates systemd differently in different levels. This creates behavioral differences in systems and their systemd installations.
IOW, systemd didn't reduce fragmentation to the level people wanted them to believe.
Moreover, pulling from that 20+ experience with Linux systems, their behavior was already this consistent since forever. RedHat had a wrapper called "service" on top of "/etc/init.d" system, but the scripts were portable. Some config files unrelated to systemd/init.d were in different places (and still are). e.g. apache/httpd, /etc/network/interfaces vs. /etc/sysconfig/network-scripts, etc.
So, the defragmentation by systemd is an illusion. Since forever, I had to either look for /etc/{redhat,debian}_version or type apt/yum/dnf and press enter to see which distro I'm on. The rest is basically the same, across hundreds of installations.
> Both desktop environments pull in different directions in appearance only.
Nope, no. KDE has a dozen toggles for every feature, half of which don't make sense. It has OK/Cancel/Apply, as if Windows 95 was peak UX. Konsole uses the same color theme regardless of whether you pick light/dark; I've tried to hack together a script, but the CLIs/APIs don't make sense: you have to detect the light/dark preference by parsing the current system theme name. Good luck if it's something like some variation of "PurPurDay"/"PurPurNight". I also couldn't find a way to get programmatically notified about the theme changing - but KDE still shoves useless notifications in my face, like "button X was removed from the panel", which you have to disable one by one. KDE can't tell "powerful" from "overwhelming".
Gnome has a top bar. It has three buttons, the rest is wasted space. The "dock" can't be moved to the left/right screen edge - more precious vertical space taken away. It has no desktop icons, even though there's a "Desktop" folder in Nautilus. It can't set up different screen dim/lock timeouts on battery vs AC. Meanwhile I was watching a PR[1] that would add iCloud integration to system accounts - it got rejected, because "users can just set up IMAP/SMTP". Gnome can't tell "simple" from "simplistic".
I could go on, but please use macOS for a week, and try to draw a fair comparison.
[1]: https://gitlab.gnome.org/GNOME/gnome-online-accounts/-/merge...
> KDE team works their heads off to be compatible with GNOME [...].
Here we go. In the other thread, I was arguing that "competition" in this space is just wasted effort.
> So, the underlying standards of many Desktop Environments are the same [...].
So if I want to write a native application, which toolkit should I target? On macOS, it's Cocoa. On Windows, it's "whatever Windows is doing in this release", but at least I can still run programs that weren't touched since 1997.
On Linux, I have a choice of plain Qt vs KDE, plain GTK (2/3/4) vs Adwaita, and so on. Apps written with one look and behave out of place when used with the other. Every time glibc breaks something, the app needs to be rebuilt. OK, so I want to provide users with a self-contained package. The choices are Flatpak, Snap, and AppImage, all of which are built differently. I could go on.
> Another notable example is, every KDE release comes with identical GTK themes for flagship Qt theme [...].
Design is not how it looks, it's how it works.
> Moreover, pulling from that 20+ experience with Linux systems [...].
Yeah, I've also used Linux on the desktop for 20 years. Then I tried macOS for a month. Then tried OS X 10.5 for a comparison. I remember considering a MacBook in 2007, I regret not getting one back then.
> Since forever, I had to either look for /etc/{redhat,debian}_version or type apt/yum/dnf and press enter to see which distro I'm on.
I've inherited and managed a heterogenous fleet of Debian, Ubuntu, Fedora, CentOS, OpenBSD, etc, (many on different releases), and had to write a lot of Ansible roles by hand, because few existing roles could handle all of these. The quirks and hacks only piled and piled. It's insane.
/rant
How is it not modular? Yes its all in one repo, but most of the parts are modular. Every distro adopts different parts of it.
Why doesn't someone make a modular systemd? Honestly I think a lot of systemd should be moved into the services its overriding or controlling.
I’ve used various forms of S6 init, you might take a look. It has its own scripting called execline, which is good, but you can also just use any shell. It doesn’t do everything systemd does but it’s quite small and very modular. https://skarnet.org/software/s6-linux-init/why.html
I like the general idea of execline, but I feel like it relies too much on chaining. It's a bit like Lisp, but without any parens - it hurts readability.
It went from "shells have too much syntax, which creates security issues" straight into "forth with blocks". My idea: runit already provides chpst, which folds things like cd, chroot, nice, etc into a single command; so let's put it all into a block with named parameters. Like this:
Maybe it could draw a bit from rc, which has actual lists as a first-class data structure.I'm not exactly sure about the syntax, but that's the general direction.
Execline is available but not in any way required to use S6.
I've used Solaris SMF back in the day, which is what systemd largely copied, but badly.
The major difference is, in SMF you specify services, their dependencies and whatever else you need. Then SMF looks at the whole desired system, computes the dependency tree and boot order, validates everything and writes a boot plan that can be executed in a static and deterministic fashion. You can look at the plan, execute that exact same plan each boot. You therefore know that if the system came up once using that plan it will again.
Whereas systemd does everything on the fly, the boot order is computed anew with each boot, all the services on the same tree level are started at the same time and just have to fight it out. Thus the actual boot order varies greatly with each boot and will fail randomly, wait randomly (hands up: who wasn't annoyed by the systemd-rememberance-minute-and-a-half popping up like an inopportune windows update?) and will be hard to debug because you just cannot reproduce anything that lead to that. Your boot will be littered with 'wait-for-something' units, 'sleep-for-10s-because-that-magically-makes-it-work' and additional dependencies to maybe clean up the tree which the maintainers forgot about. Sometimes intentionally because "systemd is about making it boot fast, and it works on my laptop".
It isn't all bad, but for systems that I need to boot and shutdown reliably, systemd is a step back even over init scripts because of the nondeterminism. You can "fix" things by just watchdog-rebooting when the system doesn't come up, and the downtime will just be a little longer. But it is annoying. And it is a design bug that will never be fixed.
Systemd has more in common with launchd than SMF to be fair. SMF very much is a product of another time. Completely fine when we were dealing with workstations whose configuration was mostly static and not expected to change but very much unfit to the environment systemd operates nowadays.
Systemd uses a dependency graph by the way. There is little magic. Provided you have properly setup what depends on what, everything will boot just fine and it’s somehow resilient enough than even if you messed up somewhere there is a good chance it manages to bring everything up after some hiccups.
OTOH figuring things out at runtime seems to be how it paints itself into corners and just... stops. I'm not going to pretend that sysv was good, but when a sysv box booted, it came up, and when I told it to shut down, it died on the spot; there was none of this sitting staring at it twiddling its thumbs. On the bright side, these days things do eventually time out; I don't miss CentOS 7 freezing forever on shutdown.
Regarding "there is little magic", Poettering repeatedly praises socket activation in his blog. It's not a new idea, but an old one that fell out of favor because it kinda sucked (remember xinetd?). It's about pretending that a service is running when by providing its listening socket, then starting it when the first client connects.
It has some problems: it's more complicated, it can actually slow things down compared to starting a service as soon as possible, and it has new error cases because of course "pretending that a service is running" doesn't have exactly the same effect as actually having the service running.
Socket activation is entirely optional and is not what I would personally call "magic".
I don't think "pretending that a service is running" is a fair description of what it does however. It's just buffering. It has advantages in some case which is why it exists. It doesn't "kinda suck".
Entirely optional, but widely used. An advertised advantage is that service A that relies on B can start before or concurrently with B. Calling that "just buffering" is really not accurate.
You are not crazy. The fact that we managed to make a livable space in and pull electricity and utilities into the shanty-town-systemd we all live in nowadays is not proof of great design.
From my own experience the alternatives are fine as long as the distro maintainers put on the effort to make them work.
I've used Devuan and others with runit, sysV, s6 and openrc. And honestly, openrc is the better alternative from an user perspective.
Nevertheless, it is hard to argue against systemd when the critical mass is there, and you run into eventual problems.
Still, I find alternatives such as runit and openrc easier to reason about than systemd. Perhaps this is due to the need to fully grok them, so you're able to fix issues yourself. Contrast that with systemd, where there are plenty of drag and drop solutions out there, so that requirement is lessened. You could also argue the real reason is systemd's complexity, but personally that doesn't feel like the core reason.
Either way, it's good that there are alternatives. I like how heterogenous the whole Linux ecosystem can be, it helps mitigate large scale security issues.
> Either way, it's good that there are alternatives. I like how heterogenous the whole Linux ecosystem can be, it helps mitigate large scale security issues.
IMHO all it does is drive fragmentation and marginalisation. Either your alternative is successful enough that third parties are compelled to care about it (making the integration work on their part more painful, which is often "solved" by using Electron); or it isn't, so you have to bring in your own integration (aka glue).
Sorry but Ballmer was right: developers, developers, developers. Users add value, developers multiply it.
I see your point, fair if you prefer it.
But I don't see it as fragmentation, but as competition. The better idea tends to prevail, and it's nice that the environment allowed alternatives to even try :)
hell, systemd itself happened as a esult of this environment :)
Some competition is OK. For example, similar programming languages (e.g. Rust and Zig) - they inspire each other, fit slightly different niches, and there's plenty of space for both.
Experimentation is OK. DNF started as a yum fork, with a different solver algorithm. IMHO yum should've just adopted the algorithm, but Fedora/CentOS ended up switching. That's mostly fine (people had to update docs, retrain their muscle memory).
The level of "competition" we see in package managers, desktop environments, initrd builders (!), resolv.conf managers (!!!), is just wasted effort. There's no "pick the winner", it's "everyone is a loser".
<https://tailscale.com/blog/sisyphean-dns-client-linux>
DNF didn't start as a competitive fork of Yum, it started as the Python 3 rewrite of Yum from the same developers. It was really just Yum version 4, using a different name so it could be installed in parallel with Yum for an extended period of time. Yum changing algorithms was never on the table, as that Python 2 code had to get phased out. Fedora, CentOS, and RHEL switching to it was always the plan.
We made heavy use of systemd and its related components on a fleet of a few thousand edge devices and it worked great for us. Occasionally, we would run into bugs or confusing behaviour and have to dig through GitHub issues, but that was far outweighed by the problems it took care of for us.
systemd does a lot more than just init. You can preach about whatever alternative inits all you like, but they don't do even 10% of what systemd does.
> systemd does a lot more than just init. You can preach about whatever alternative inits all you like, but they don't do even 10% of what systemd does.
I didn't mean to preach about alternative inits.
My personal criticism of systemd is that it tries to take over everything, limiting the choice.
I want to have a choice, that's all. And systemd doesn't do anything to help there.
Can you give concrete examples?
systemd-resolved: can be disabled and replaced with your DNS resolver of choice
systemd-timesyncd: can be disabled and replaced with your NTP daemon of choice
systemd-networkd: can be disabled and replaced with your own networking scripts
Modularity would be to be able to use, say, systemd-logind or systemd-udev without using systemd as your init, as there is no tangible or important dependency on the use of systemd as init for those functions.
On a fully-systemd system you can opt out of systemd features. But you cannot opt in to systemd features on a non-systemd system.
> there is no tangible or important dependency on the use of systemd as init for those functions.
How do you catch device events on very early startup (e.g., between start of pid1 and when processes start to fire?) if the thing that manages the devices isn't integrated with pid1 at a deep level?
logind does not "manage devices", it just provides an API for your display server to open privileged I/O devices and get a file descriptor, and have these file descriptors revoked later.
The difference with a plain `open` is that logind will allow only one process at a time to have access, and only for sessions that logged in physically (tracked using PAM), and does not require the process to have direct file access (which could enable key-logging). It will also revoke the file descriptors when switching sessions.
logind's other random features like dock and power button triggers can be done whenever logind is running and ready - the underlying API reports levels and are not timing critical.
The only thing logind really uses at runtime from systemd is temporary storage of file descriptors that it can get back after restart for graceful upgrades.
Projects like elogind and eudev exists that painstakingly strip out the systemd portions while retaining the functionality, but it's an uphill battle to constantly redo every time there's a change.
Thank you, that's what I was trying to say, but worded in a vastly superior way :-).
I am telling you that it is difficult to use systemd subcomponents without running all of systemd, and you give me example of running most of systemd and replacing a few subcomponents.
Can you run those 3 standalone?
>I am telling you that it is difficult to use systemd subcomponents without running all of systemd
Actually it's the first time you have stated that, but I'll bite.
How do you propose that would work, and why would you even want to do that? The entire point of having these modules is for them to plug in to core systemd, which enables a lot of magic that would otherwise require an amalgam of shitty shell scripts. If you don't want to use systemd, why not just use chronyd for NTP, dnsmasq for DNS, and so on? Your problem is completely imaginary.
> Actually it's the first time you have stated that, but I'll bite.
Fair point, I said it in another comment :-).
> How do you propose that would work
I don't. I merely state the fact that I don't like the way it's going with systemd, which is that many projects now assume that you have systemd and depend on it. And if you don't have systemd, you're screwed.
Similar to projects that assume that you have Ubuntu and depend on Ubuntu specificities.
And again I'm not saying that it should be illegal; just that I don't like it.
I haven't looked into this but most of these have D-Bus interfaces no? Why couldn't you use them without systemd.
Your last paragraph is why most of the people who don't like systemd don't like it.
People who don't like it can call me back when they graduate beyond ricing their i3 desktop and run a ~5k node server fleet, and then tell me how they did it without reimplementing everything systemd already does.
> run a ~5k node server fleet,
That is why RedHat wants it. What does it help on the laptop that I am writing this?
And what happens when you upgrade that fleet (unless you're doing clean installs) and sytemd-resolved is not resolving anything? (Just the most recent thing I've had)
Well the key is to always do clean-installs and join the fleet and decommission the old machines. Machines are essentially immutable and disposable, so upgrades are done with full machines. And if anything hardware goes bad on whole or sometimes half racks.
I'm very curious about specifically what is it that it does for you that is so amazing, besides init that you can't live without?
Not the person you're asking, but I'm curious what you mean when you say "the alternatives".
I've used runit, openrc, briefly upstart (what a trashfire that was), and I don't miss a single thing about them while writing systemd unit files.
I've used GNU Shepherd on Guix, and honestly, meh, systemd feels way more pragmatic.
I've also used systemd-networkd as well as NetworkManager, and feel like they both still have a few warts, but overall prefer systemd-networked on servers.
I've used grub, refit, and systemd-boot, and gotta say systemd-boot is my preference there too.
I haven't really used systemd portable services, but I've used docker and don't like it, and my overall impression is I'll probably like portable services more.
Are there other alternatives I have to use before I'm allowed to have an opinion? What alternatives have you used which made you feel so enlightened?
Personally, it's not that the alternatives are so awesome.
It's when i meet against some awful design choices in systemd, like how it decides to wait forever rather than hard fail, and by wait forever, i mean wait for 30 seconds, no, 1 minute 30 seconds, no 5 minutes... you get it.
Or you can easily lock yourself out when there's a typo in /etc/fstab.
Or the hardcoded 5 minute timeout for sysv-generator: https://github.com/systemd/systemd/blob/main/src/sysv-genera...
God forbid you used cgroupsv1 for anything when poettering unilaterally decided to punish everyone with a 30 second delay for using it.
If you stray off of the intended opinionated path, be prepared for the least user-friendly experience linux has to offer. You might even get some abuse from the developers if you try to ask for help, but they've probably toned it down since CoC became a thing (I hope).
God forbid you used cgroupsv1 for anything when poettering unilaterally decided to punish everyone with a 30 second delay for using it.
Intentional time-delays are a nice middle ground. Yes, it is annoying to the users who've ignored all prior deprecation warnings, but it's better than those same users being "surprised" when support is removed entirely. Maybe you'd consider it less of a "punishment" if systemd dropped support for cgroupsv1 outright instead of inserting a time delay, I believe the opposite.
> What alternatives have you used which made you feel so enlightened?
I didn't mean to seem like I feel so enlightened.
My experience is that when not using systemd, I regularly get screwed by what I would call the "imperialist culture" of systemd, for lack of better words: it doesn't feel like you can relatively easily extract subcomponents of systemd and use it standalone. And more and more projects depend on a particular subcomponent of systemd, meaning that they require systemd.
So it's not so much that I believe systemd is technically worse than the alternatives. It's more the philosophy that I don't like.
Blatantly not the person you're asking, but once you go past basic service management trying to do it just about anywhere else is an absolute and total nightmare. People who have major problems with systemd more than anything just ever seem confused, often vastly overestimating systemd's actual scope.
The reality to many is its the simplest simple way to handle some very complex tasks, and the vast majority of it is still all so very very much optional even in the most 'systemd heavy' distros.
Is ParticleOS also a useful everyday driver for Linux enthusiasts who aren't systemd developers? Let's say I got bored with Gentoo and I want to try something new. Is ParticleOS something that a technical user would enjoy at the current stage?
I think you'll get the most out of this at the moment if you're interested in actively contributing to systemd. There's lots of work left to be done to make this usable which will just be annoying if you aren't interested in fixing some of this stuff yourself.
Of course if you're interested in doing small prs and such to improve things then I think it could be very satisfying. You can always join the mkosi matrix channel to chat more about this stuff.
Nix would let you do just this already with potential binary cache and reproducibility.
I’m surprised it’s not more widely used for embedded. Maybe due to larger images? That’s a solvable problem.
Maybe best of all it’s fast. Yocto is painfully slow.
I'm not sure nix is aimed at the same kind of things here. ParticleOS seems to be a logical continuation of the effort that systemd has been putting into TPM2, boot security, signing, and immutability.
> Currently both arch and fedora are supported distributions.
Does this mean that we can now have an actual immutable Arch-based distro?!
I love Silverblue, best distro ever, but the day I can have immutable Arch Linux I’m outta here! I truly despise rpm and all the fedora machinations required to build a custom package.
Yeah I've been running the Arch variant of this on my personal laptop for a while now. Of course no stability guarantees with this for now since you'll be running systemd from source.
How does the update process work? Do you run pacman -Syu and then bake the result into the next image with mkiso?
mkosi runs pacman for you and then packs up the result as a disk image. It also builds a unified kernel image and does a bunch of signing. The new /usr partition and UKI are then installed with systemd-sysupdate.
How have your experiences with this setup been so far? Any major pain points?
Building the next update locally is slow because erofs has to compress the entirety of /usr on my rather old laptop and that takes a while.
Aside from that, I'm using flatpak for firefox and for some reason it takes absolutely forever (like > 15s) to start firefox on my machine, still need to look into why that happens.
Aside from those it's very usable. But of course it's running systemd from source so I'm not going to actually recommend anyone to run this who is not a systemd maintainer until we iron out the kinks.
Thanks!
The slow startup times have usually been an xdg-desktop-portal issue for me in the past, might be worth looking into.