cffi VS dense-arrays

When you want to do anything that breaks the nice bubble of your Lisp image, you might want to know a bit about your operating system's programming interface. This will come in handy if you ever need to wrap a library with CFFI. There are some things that are pretty inconvenient as a rule (like dealing with any protocol that uses network byte order), but if you stay within the bubble of your Lisp image, you won't really notice them.

I'm just posting the work that I have done over the last year or so on CL-Objc. I'm still blocked from better support (e.g., passing structs by value for frameworks like UIKit). I just wanted to post what I have done online for others interested in the work or motivated to collaborate on this.

Good morning ladies and gentlemen, I have been waiting on some feedback for PR in CFFI. This feature is blocking me from reviving CL-OBJC. Any help will be appreciated. Thank you in advance.

Is that actually https://github.com/cffi/cffi/commit/33970351e71bb5f12ba56fc40270089e948ae112 ? I.e. after loading cl+ssl. (Although Hunchentoot does not interrupt threads)

Maybe access the pins using CFFI, https://github.com/cffi/cffi package and one the libraries mentioned here? https://www.bigmessowires.com/2018/05/26/raspberry-pi-gpio-programming-in-c/

FTR, on my system QL fetches CFFI 0.23.0 and the fix/error I'm talking about is https://github.com/cffi/cffi/blob/master/src/libraries.lisp#L106 and seems to have been added iin this PR https://github.com/cffi/cffi/pull/173/commits/263b38f4f2600dbacde8f2b313620c35a563c6df so the fix should be in CFFI 0.24.0 released 24 March 2021.

FTR: this is the PR https://github.com/cffi/cffi/pull/173/commits/263b38f4f2600dbacde8f2b313620c35a563c6df

C: Alternatively (more difficult) you could try to wrap the underlying C layers of either of those mentioned under Python with CFFI. The C-based game engine, Raylib, is also wrappable this way. I finished a super cool walking simulator in CL with that, but it is more tedious than the others since raylib is really barebones.

I feel inspired to start Lisp after being disappointed with the "open" source scene of 2021. I'd rather pay LispWorks a yearly fee and be left alone than dealing with unbalanced people in the Python space. The free Lisp implementations also look somewhat isolated from the ideological wars.

However, a C interface is required. Is this one the recommended solution? Is it really portable?

https://common-lisp.net/project/cffi/

What is the speed compared to a Python C extension? Are implementation-specific C interfaces faster (I guess they are)?

Sorry for so many questions, but these can usually only be answered by people who have actually used the interface.

*I have not; I have a couple of WIP/alpha-stage libraries like dense-arrays and numericals that could be useful; once I find the time, I want to think about if these or its dependencies can be integrated into the existing libraries including antik mentioned by awesome-cl.

Personally, I've been relying on the stream-based method using py4cl/2, mostly because I did not - and perhaps do not - have the knowledge and time to dig into the CFFI based method. The limitation is that this would get you less than 10000 python interactions per second. That is sufficient if you will be running a long running python task - and I have successfully run trivial ML programs using it, but any intensive array processing gets in the way. For this later task, there are a few emerging libraries like numcl and array-operations without SIMD (yet), and numericals using SIMD. For reasons mentioned on the readme, I recently cooked up dense-arrays. This has interchangeable backends and can also use cl-cuda. But barring that, the developer overhead of actually setting up native-CFFI ecosystem is still too high, and I'm back to py4cl/2 for tasks beyond array processing.

Currently I have put successfully this to use at dense-numericals - which I created over dense-arrays after finding CL arrays to be not that suitable, as compared to numpy or julia. Now, dense-numericals relies on passing the array pointer to C functions. However, IIUC, this runs into issues for what if the GC moves the arrays while the computation is still not done; is this worry valid? I think I ran into this while running multithreaded tests on CCL, ending up in segfaults.

Shameless unstable plug: I think it should be possible to provide type checking with a different backend that does not upgrade the types at https://github.com/digikar99/dense-arrays - the backend things are themselves unstable though.

In semi-production, ideally the problems are best represented using state diagrams, but I don't see a way to comfortably represent graphs in textual formats. The best I see is list of lists, which doesn't feel significantly better than the spaghetti code it currently is (for instance this and this - but these are just about one function each in a larger system, so not totally worth a DSL, unless there existed a defacto state-diagram DSL which everyone could be expected to know.