DeepMenu VS EffectivePower

I've been working on the menuing side [1] based on crossing Fitt's Law with Huffman trees. But, don't know the constraints for ALS.

Hopefully, whomever takes this on doesn't take the standard Accessibility approach, which is adding an extra layer of complexity on an existing UI.

A good friend, Gordon Fuller, found out he was going blind. So, he co-founded one of the first VR startups in the 90's. Why? For wayfinding.

What we came up with a concept of Universal design. Start over from first principles. Seeing Gordon use an Accessible UI is painful to watch, it takes three times as many steps to navigate and confirm. So, what is the factor? 0.3 X?

Imagine if we could refactor all apps with a LLM, and then couple it with an auto compete menu. Within that menu is personal history of all your past transversals.

What would be the result? A 10X? Would my sister in a wheelchair be able to use it? Would love to find out!

[1] https://github.com/musesum/DeepMenu

Am experimenting with a new menuing idiom with SwiftUI [1]. It is both delightful and infuriating. Am delighted by autolayout. But, way too much semantics for observers. To simplify, I intentionally avoided structs, for view models, and resorted to classes.

I hope to create a package and use it for metal based visual music synthesizer. But, those complaints about playing nice with UIKit? Am rather worried.

[1] https://github.com/musesum/DeepMenu

Pretty recent, I check it every once in a while to see what’s going on. Of course, with macOS generally most of the changes happen around WWDC ;)

SwiftUI on macOS is currently ok for a few usecases. One is little document apps. Emphasis on little, this is going to be things that wrap an image or a text file or something. The API is reactive so it’s not really designed for complex/partial file formats. Another good usecase is just pure drawing/animation. This code will look identical on macOS and iOS and is very easy to write with SwiftUI. Finally, the last thing that mostly works is just lists of content. Not table views, those aren’t very good yet. I’m talking about Twitter-like content, as in a media viewer in list form.

What doesn’t work well is anything that has to do with controls (either they look very ugly, the control doesn’t exist, or SwiftUI has no concept of some significant portion of the control). If you want to do like focus loops or right clicks or anything complicated, forget it. (Some of these are getting fixed piecemeal, but AppKit has like a thousand little things that make good apps and it’ll be a while before they are all fixed.)

By way of example, I actually have some small apps that show this off well. https://github.com/saagarjha/EffectivePower is a little view-only document shoebox style app. It’s 100% SwiftUI, has one main screen that is a hand-drawn graph, and some auxiliary UI. This makes it about as good as it could be for using the framework. Making the graph was a delight. Making it perform well was a little bit of work but not too bad. The document abstraction mostly works except SwiftUI expects me to be able to create new documents and I’m just a viewer app so I need to crash if you try to make a new document. The sidebar is a List, which is fine, but I actually wanted it to be a Table. However Table selection asserts right now if you update its selection faster than the NSTableView that backs it animated. Of course in SwiftUI there is no concept of animation duration so you can’t really stop this from happening. You can also zoom the graph in the app. On Monterey if you tried to zoom a couple of times the system magnification gesture just stopped working silently. On Ventura I believe they have fixed this, or at least it isn’t trivially broken anymore. This is common for SwiftUI.

By way of contrast, https://github.com/saagarjha/VirtualApple is also pretty simple but it’s 100% AppKit. I had originally written it in SwiftUI but it’s just not possible to make it be a good Mac citizen. It is “document based” but because it deals with multi-gigabyte VMs I can’t actually let it do the default document based behavior, which includes making hidden revisions whenever the files update (which means you get a dozen secret copies of the VM). In AppKit I can just turn this all off and still look identical to any other system app. (Caveat: if I want the recent files menu and don’t want to use a storyboard, this is actually not available even in AppKit without SPI). Also, I have a little configuration screen which is all buttons and checkboxes. AppKit lets me align it precisely how I want so it looks nice. In SwiftUI it would look pretty awful.

I just finished writing a little Mac app in SwiftUI for my personal use yesterday: https://github.com/saagarjha/EffectivePower (https://twitter.com/_saagarjha/status/1528937319171887104 for a slightly out-of-date screenshot). Here are my thoughts.

The good:

* Making all the graphs (they're drawn by hand!) was very nice and easy. The speed at which I could iterate with them was incredible, and I could just move code around and lay things out and move them between moves in a way that AppKit (or UIKit) would not allow, at all.

* I could keep the code very clean, with each component being very specific and isolated. Plumbing bindings through made things very natural and I can imagine taking the code I made there and just plopping it into another project as-is.

* The result is actually kind of nice, IMO? There's a lot of focus on whether SwiftUI works and stuff but not much focuses on what the end result is. I had an app in my mind that I wanted to make, and it was mostly possible to make it in SwiftUI. In some places I was pleasantly surprised that things I would have been hesitant to try before (slider in a toolbar!) "just worked".

The middling:

* I initially supported macOS 12.0 only. Someone asked me to backdeploy to 11.0, which was a little painful mostly because Material didn't exist back then and neither did support for initializing a color from a NSColor. I did kind of a lazy stab at it and the end result being fairly simple, but took about an hour to write: https://github.com/saagarjha/EffectivePower/blob/main/Effect.... If I had to support 10.15 I think I would honestly rewrite large parts of the app in AppKit, maybe keeping just the graphs as view representables.

* My data model has tens of thousands of elements. Ensuring the "reactivity" didn't cause a bunch of things to be recalculated when they shouldn't was a bit of a challenge. The place I have it now is very nice (I have them defined in such a way that they will never update unless they need a redraw) but this definitely does not come "for free", you'll find out about it after profiling and need to figure out how to fix it.

* I wrote simple versions of things that don't seem to exist in SwiftUI but AppKit provides for free, such as magnification bouncing. It was like three lines of code to get something that seems reasonable, but with SwiftUI I'm never sure if this is a "we just didn't add it yet" thing or a "oh this is so simple in the framework that you should just write it yourself".

The bad:

* Things are broken and I don't know why. If you use magnification gestures the callbacks stop getting called. No indication why. I have some commented out code that would've used a table in the sidebar, but I had to use a List instead because it seems like SwiftUI does not properly update the table and it crashes with an assertion.

* Documentation sucks but that's nothing new. There's a lot of things that work but you need to be clever at arriving to getting to that point. It's a fun challenge for a toy project but for a production thing I can see this being super frustrating.

* If you mismatch a type somewhere the compiler is just going to time out rather than telling you what is wrong. Thankfully you can just go through and comment out large parts of the app to reduce the scope of where the error is coming from, but the fact that this is necessary is kind of annoying.