Drone-ESP32 VS dRehmFlight

Project Source for Drone Controller

Since you mentioned programming, one other thing you should consider is the firmware. FPV drones typically run the betaflight firmware. betaflight is fairly mature, feature-rich and is aimed at racers and freestyle pilots. If you’re looking more to tinker with programing, you could start with this instead of a more standard flight controller: https://github.com/nickrehm/dRehmFlight

I have similar aspirations. No books, but if you want an example of someone who’s done that before, check out drhemflight. It’s not fully autonomous, but it’s a modular flight stabilizer running on teensy, which uses arduino IDE

There's several flight control systems on Github, such as Ardupilot, for example. I wouldn't undertake this without checking out what's already out there, all the way down to simple stabilization systems.

To add to this, most of the time one doesn't need to hack on the drone firmware itself. Just configuring/hacking the mission planner alone would go a long way: https://ardupilot.org/planner/

But if you're interested in more readable codebases without too much legacy headaches:

https://github.com/BossHobby/QUICKSILVER (for indoor drones mostly)

And for even less functionality but a good learning tool kind of codebase:

https://github.com/nickrehm/dRehmFlight

dRehmFlight is about the simplest a flight controller can get (in terms of code readability).

This is actually my own flight controller code running on a teensy 4.0 microcontroller, coded in arduino: https://github.com/nickrehm/dRehmFlight