[{"_1":2,"_3":-5,"_4":-5},"loaderData",{"_5":6},"actionData","errors","routes/hashtag",{"_7":8,"_9":10,"_11":12},"articles",[13],"authors",{},"tag","nspawn",{"_14":15,"_16":17,"_18":19,"_20":21,"_22":23,"_24":25,"_26":27},"content","# How I put NixOS on my UDM (trashcan model) router\n\n![A rare cursed fetch!](https://despera.space/assets/img/cursed_udm_fetch.png)\n*a rare cursed fetch!*\n\nContent also available on [https://code.despera.space/iru/htdocs/src/branch/main/notes/UDM-NIXOS.md](https://code.despera.space/iru/htdocs/src/branch/main/notes/UDM-NIXOS.md)\n\nReally it's just a running NixOS on systemd-nspawn thing.\n\nThe UDM product line basically runs on Linux kernel and userland. It is a \nsurprisingly normal device that allows you to SSH and run commands. It even has \napt and systemd services installed. The only catch being that for the most part\nthe file system structure is immutable with only a few exceptions like /data and\n/etc/systemd. Previous versions even had the Unifi services running on a podman\ncontainer. On recent versions of the firmware podman was phased out but we got\nsomething that resembles a more complete system structure as opposed to a\nbusybox-like system.\n\nSo basically its some kind of Debian-based Linux running on a headless ARM64 \ncomputer. Can we install and run stuff? Yes! In fact projects like \nhttps://github.com/unifi-utilities/unifios-utilities publish scripts to run \ngeneral purpose programs and configurations on UDM. Be aware however that\nfirmware upgrades might wipe the persistent data storage so don't put anything\nin there that you don't want to lose and preferably keep scripts so you can\nsetup again after having its flash storage nuked by a major update.\n\nI have the base UDM model. The first with the pill format that has been \naparently replaced by the UDR. The UDR seems to have more features like Wifi6,\nbigger internal storage and even an SD card slot meant for vigilance camera\nfootage storage but comes with a weaker CPU in comparison with the original\nUDM base. As far as I know the rack mountable models follow the same OS and\nfile system structure.\n\n\n## Okay but why?\n\nI'm gonna leave this to your imagination on why would you add services to your\nproprietary router applicance. To me its the fact that I don't really like\nrunning servers at home and I'm ultimately stuck with this router so why not\nput it to work maybe running a static webserver or something silly like Home \nAssistant. The truth of the matter is that I can't just leave things alone.\n\nAnd if you can run Linux why would you run something that is not NixOS? Thats\ncrazy and it doesn't make sense.\n\n## How do we root the UDM? What kind of jailbreak do I need?\n\nNo.\n\nYou enable SSH from the Controller UI, log into it as root with the password you\nset to the admin user. You just waltz in and start installing and configuring.\n\n```\n# apt update && apt install systemd-container \n```\n\nThats it. Kinda. The complicated part is modifying the programs to write into\nthe persistent data directories while also making sure your stuff starts on \nboot and doesn't get wiped on minor firmware upgrades.\n\n## Building the NixOS root image.\n\nMight want to read first: [https://nixcademy.com/2023/08/29/nixos-nspawn/](https://nixcademy.com/2023/08/29/nixos-nspawn/)\n\nWe need a NixOS tarball image. TFC's https://github.com/tfc/nspawn-nixos\ncontains the flake to build such an image and also publishes artifacts for AMD64\nbut not ARM64. I guess you could build this from an AMD64 machine but I haven't\nlooked into building a cross platform environment (didn't needed to compile\nanything though). I have a recent macbook with UTM so I just downloaded one of\nthe default Linux virtual machine images from the UTM page and installed the \nNix runtime over the OS.\n\nMake sure you have git and curl installed. \n\n```\n$ sh <(curl -L https://nixos.org/nix/install) --daemon\n```\n\nYou need to start another terminal session. \n\n```\n$ git clone https://github.com/tfc/nspawn-nixos\n$ cd nspawn-nixos\n$ nix --extra-experimental-features 'nix-command flakes' build .\n```\n\nOptionally you could try to edit the configuration to generate an image with\neverything you need. In case you need something like Home Assistant, some \ncompilation might be necessary and although I haven't tried compiling code on\nthe UDM I suspect it might be a painful process due to CPU performance and\nspace constraints. Here is an [example with Home Assistant, Caddy and Tailscale](https://code.despera.space/iru/nspawn-nixos/src/branch/main/configuration.nix).\n\nThe image will be available under\n`./result/tarball/nixos-system-aarch64-linux.tar.xz`. Use scp to send this to\nthe /data/ directory of the UDM.\n\n## Installing the image\n\nFirst we create the folder structure:\n\n```\n# mkdir -p /data/custom/machines\n# ln -s /data/custom/machines /var/lib/machines\n```\n\nUnder normal circunstainces by now you would just run \n`machinectl import-tar /data/nixos-system-aarch64-linux.tar.xz `\nhowever the version of tar that is present in this system doesn't really like\nthe resulting tarball image. It will yeld errors like `Directory renamed before\nits status could be extracted`. \n\nThankfully we can install bsdtar through `apt install libarchive-tools` however\n`machinectl import-tar` is hardcoded to use the tar command. Adding a symlink\nfrom `/usr/bin/bsdtar` to `/usr/local/bin/tar` won't work since some parameters\nare used that are not supported in bsdtar. You could try writing a wrapper shell\nscript but just unpacking the tarball directly was sufficient.\n\n```\n# mkdir /var/lib/machines/udmnixos\n# bsdtar Jxvfp /data/nixos-system-aarch64-linux.tar.xz -C /var/lib/machines/udmnixos\n```\n\nLets start the container.\n\n```\n# machinectl start udmnixos\n# machinectl\nMACHINE CLASS SERVICE OS VERSION ADDRESSES\nudmnixos container systemd-nspawn nixos 23.11 192.168.168.88…\n\n```\n\nGood. Now we need to change the root password.\n\n```\n# machinectl shell udmnixos /usr/bin/env passwd\nConnected to machine udmnixos. Press ^] three times within 1s to exit session.\nNew password: \nRetype new password: \npasswd: password updated successfully\nConnection to machine udmnixos terminated.\n```\n\nFinally we can login into the container.\n\n```\n# machinectl login udmnixos\nConnected to machine udmnixos. Press ^] three times within 1s to exit session.\n\n\n<<< Welcome to NixOS 23.11.20240115.b8dd8be (aarch64) - pts/1 >>>\n\n\nnixos login: root\nPassword: \n\n[root@nixos:~]# \n```\n\nWe haven't finished yet. By default the network is set to its own container\nnetwork. We also don't have a DNS resolver configured. You can leave that\nsession with CTRL+]]].\n\nhttps://www.freedesktop.org/software/systemd/man/latest/systemd-nspawn.html#-n\n\n```\n# machinectl stop udmnixos\n```\n\n## Networking and Persistence\n\nThe first thing that needs to be addressed is the DNS configuration. The default\nsetting that copies the /etc/resolv.conf from host won't work since it points to\nlocalhost. Either install resolved, netmask or set a static DNS config.\n\nAs for the network method we have some options here. \n\n- [Run using the default network stack and map ports to the container](https://www.freedesktop.org/software/systemd/man/latest/systemd-nspawn.html#-p). \n- Run using something akin to --network=host where the container has full access to the host network.\n- Give the container its own independent interface through a bridge.\n- [Give the container its own independent interface through macvlan](https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container#step-2a-configure-the-container-to-use-an-isolated-macvlan-network).\n\n### Using --network-veth and port mapping\n\n```\n# mkdir -p /etc/systemd/nspawn\n# cat > /etc/systemd/nspawn/udmnixos.nspawn < /etc/systemd/nspawn/udmnixos.nspawn < /etc/systemd/nspawn/udmnixos.nspawn <>>\n\n\nnixos login: root\nPassword: \n\n[root@nixos:~]# ifconfig \nhost0: flags=4163 mtu 1500\n inet [redacted] netmask 255.255.255.192 broadcast [redacted] \n inet6 [redacted] prefixlen 64 scopeid 0x20\n inet6 [redacted] prefixlen 64 scopeid 0x0\n ether 92:01:4c:a7:a1:7b txqueuelen 1000 (Ethernet)\n RX packets 2415 bytes 611986 (597.6 KiB)\n RX errors 0 dropped 0 overruns 0 frame 0\n TX packets 61 bytes 5337 (5.2 KiB)\n TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0\n\nlo: flags=73 mtu 65536\n inet 127.0.0.1 netmask 255.0.0.0\n inet6 ::1 prefixlen 128 scopeid 0x10\n loop txqueuelen 1000 (Local Loopback)\n RX packets 0 bytes 0 (0.0 B)\n RX errors 0 dropped 0 overruns 0 frame 0\n TX packets 0 bytes 0 (0.0 B)\n TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0\n\n```\n\n### MACVLAN isolation and more\n\nHere is where some custom configuration might be needed. Read https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container \nto find out how to setup custom scripts.\n\n## Persistence\n\nAs far as I verified by rebooting the UDM many times to write this note all\nconfigurations were preserved. According to [the article on nspawn-containers on the unifies-utilities project](https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container#step-3-configure-persistence-across-firmware-updates)\nalthough `/etc/systemd` and `/data` folders are preserved during firmware upgrades `/var/` and `/usr/` are not and there goes our packages and symlink. Please follow the steps on that \npage to setup persistence across firmware upgrades. ","created_at",1716148264,"id","bc01bec975057ce690f027f1bd66f204b1a165baf8335592ae515d96c869ef7e","kind",30023,"pubkey","9eeca894b2e8830e05ec86f4d119de411a2dd3abcb65525db69c486f4afe5882","sig","e4f2cecafda55043915506dfc29e5eaaa93be7522f5435fa0ca1929fff5f65af8d8483ba2bf27de15c780c9952beaf4fbdca5be2341bf43fd5ae95bab128ba50","tags",[28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44],[63,64],[61,62],[59,60],[45,46],[57,58],[45,46],[45,46],[49,56],[49,55],[49,54],[49,12],[49,53],[49,52],[49,51],[49,50],[47,48],[45,46],"published_at","1716127718","image","https://despera.space/assets/img/trashcan.png","t","Daring","Capability","machinectl","step","containers","UDM","nixos","alt","This is a long form article, you can read it in https://habla.news/a/naddr1qvzqqqr4gupzp8hv4z2t96yrpcz7eph56yvausg69hf6hjm92fwmd8zgda90ukyzqqxnzde3xccnyd3sxver2dfk5j9lzf","summary","Shoving NixOS where is not supposed to be.","title","How I put NixOS on my UDM (trashcan model) router","d","1716126032556"]