ardupilot VS dRehmFlight

They are widely used in narrow circles :)

Two of (arguably the best) open source RC aircraft flight controllers (ArduPilot and PX4) are using extended Kalman filters in their state estimators (essentially sensor fusion that provides attitude/position estimate):

https://github.com/ArduPilot/ardupilot/tree/master/libraries...

https://github.com/PX4/PX4-Autopilot/blob/main/src/modules/e...

I'm not that familiar with cleanflight/betaflight/inav scene to know what the FPV racer flight controllers use.

2. https://ardupilot.org/

Trimble also has some nice stuff, I think the BD940 is interesting. I do not know if it's recommended or not, but Ardupilot has a driver for the Septentrio GPS (https://github.com/ArduPilot/ardupilot/blob/master/libraries/AP_GPS/AP_GPS_SBF.cpp), as well for Trimble GPS (https://github.com/ArduPilot/ardupilot/blob/master/libraries/AP_GPS/AP_GPS_GSOF.cpp). So people have used these brands for drones.

For programmable open source autopilot: Arduino > Ardupilot (https://ardupilot.org).

You might also want to look at something like Ardupilot

I'd like to emphasize the reference to Ardupilot bc there is a lot of material already developed there: a lot about navigation and guidance from UAVs, and the low-level stuff. Check this out

Since you mentioned programming, one other thing you should consider is the firmware. FPV drones typically run the betaflight firmware. betaflight is fairly mature, feature-rich and is aimed at racers and freestyle pilots. If you’re looking more to tinker with programing, you could start with this instead of a more standard flight controller: https://github.com/nickrehm/dRehmFlight

I have similar aspirations. No books, but if you want an example of someone who’s done that before, check out drhemflight. It’s not fully autonomous, but it’s a modular flight stabilizer running on teensy, which uses arduino IDE

There's several flight control systems on Github, such as Ardupilot, for example. I wouldn't undertake this without checking out what's already out there, all the way down to simple stabilization systems.

To add to this, most of the time one doesn't need to hack on the drone firmware itself. Just configuring/hacking the mission planner alone would go a long way: https://ardupilot.org/planner/

But if you're interested in more readable codebases without too much legacy headaches:

https://github.com/BossHobby/QUICKSILVER (for indoor drones mostly)

And for even less functionality but a good learning tool kind of codebase:

https://github.com/nickrehm/dRehmFlight

dRehmFlight is about the simplest a flight controller can get (in terms of code readability).

This is actually my own flight controller code running on a teensy 4.0 microcontroller, coded in arduino: https://github.com/nickrehm/dRehmFlight