Linux lost and found, pt.1

I f*-ed up my kernel drivers.

I want to dedicate this part of linux lost and found to most recent screw up that happened - I f*ed up my kernel. What happened?

I am on my Fedora 41 - 6.12.4-200.fc41.x86_64, day is sunny, I am about to dive deep inside js's v8 fuzzes and escapes, connecting laptop to a usb-c docking station - and it does not show external monitors, bluh. After some web browsing (not so long one) decided to try fixing it with some displaylink modules/drivers. That did not fix it, so I removed them with dnf, but guess what - SOMEHOW, don't ask me how, kernel net drivers/modules got deleted (or didn't rollback to previous state or something like that). That got me completely offline.

Later on, figured out that bunch of other kernel drivers/modules went missing as well as I was experiencing performance drops/stutters. Couldn't connect via wireless, ethernet, usb tethering, bluetooth and only interface that was showing was local:lo one. Bummers.

Okay I kind of had an idea of what was going on, even tho frustrated because I can't continue my previously intended workflow. I was confident that this can't take me more then a few minutes to resolve, as there is probably bunch of posts online with similar situation. Not long after, minutes turned into hours, and nothing I found online helped.

Lets delve into technicalities.

How to identify this situation

Running lspci -k, we can see that adapter is recognized, but there are no loaded modules/drivers for it, same situation with lsusb -t for ethernet adapter (realtek in my case) and iphone (usb tethering).

Going inside /lib/modules/$(uname -r)/kernel/drivers/net/*, everything was wiped out, empty, not a single kernel object...

Trial and error

There was just one thing I though of, building those drivers from source. It was too much of a hustle building MT7921(my wireless card) drivers, and I didn't have ethernet adapter on me at the time, so going with ipheth (iphones usb tethering) driver made most sense.

Kernel requires libimobiledevice and usbmuxd for usb tethering - luckily those two survived my havoc. Running systemctl status usbmuxd.service we can see that upon connecting iphone, system is mounted, there were some problems with lockdown access, but chaning ownership resolved that. Anyhow, in order for our system to detect that new interface - ipheth was needed, yet missing.

I found its source code inside Linus Torvalds githubs repo, tried building it, and wohoo, I did not have 'make/gcc' installed... Went on a frenzy collecting packages/dependencies to manually install those so I can build that ipheth, and eventually gave up because those need like a billion dependencies to make it stable/work.

Luckilly fedora keeps 3 kernel versions in case kernel update, or some stupido like miself breaks stuff... Soo catching grub at runtime (esc or shift), I choose to boot on to '6.11.10-300.fc41.x86_64' and do further work on it as those drivers are intact and I am back online.

I then built that kernel object from source code of ipheth, and as I was about to push it to 6.12.x (broken) kernel, move inside /lib/modules/....../net/usb/ when a not(so bright) idea came up to my mind. Lets just grab existing kernel object from this kernel and push it over to new one, its essentially same as my compiled/signed one, but with no probability of some screwup I could've made in process.

Importing that kernel object inside 6.12.x, and running depmod to reload modules, and modprobe ipheth to load it. It fails, because from modinfo ipheth we see that vermagic is 6.11.x still, as it is built on/for 6.11.x kernel and it won't load on others (duh)...

Salvation


Fresh installation of Fedora 41 as a virtual machine, upgrading to 6.12.x kernel (mirror of my broken one), and copying kernel object of ipheth and moving it to my broken kernel - loading it, saved the day and I got network via usb tethering from my phone. After that I just ran update, and dnf picked up on all those missing pieces and after that, everything was back as it originally was.

This was (not so)fun learning experience ^^

TL;DR

Fresh installation of Fedora 41 as a virtual machine, upgrading to 6.12.x kernel (mirror of my broken one), and copying kernel object of ipheth from /lib/modules/$(uname -r)/kernel/drivers/net/usb/ and moving it to my broken kernel - loading it (modprobe ipheth), saved the day. I got network connection over usb tethering, and afterwards ran online update which resolved rest of missing modules/drvrs.

I am kinda 'proud' of this 'method' as its something I came up with on the run for this particular situation, as scraping thousands of forums for answer did not provide me with anything even alike..

Please do let me know if you have another way of resolving this situation, looking forward to it^^


Until next linux lost and found session, stay healthy, God bless y'all.

bigfella