awkward VS sqlmodel

there's a whole ecosystem in Python originally developed for high energy physics data processing: https://github.com/scikit-hep/awkward all because Numpy demands square N-dimensional array

Same technique used everywhere, here's a simple Julia pkg for the same thing: https://github.com/JuliaArrays/ArraysOfArrays.jl/blob/3a6f5b...

But Julia at least has the decency to just support ragged Vector{Vector} out of the box, and it's not that slow

Awkward Array.

Numba's @vectorize decorator (https://numba.pydata.org/numba-doc/latest/user/vectorize.htm...) makes a ufunc, and Awkward Array knows how to implicitly map ufuncs. (It is necessary to specify the signature in the @vectorize argument; otherwise, it won't be a true ufunc and Awkward won't recognize it.)

When Numba's JIT encounters a ctypes function, it goes to the ABI source and inserts a function pointer in the LLVM IR that it's generating. Unfortunately, that means that there is function-pointer indirection on each call, and whether that matters depends on how long-running the function is. If you mean that your assembly function is 0.1 ns per call or something, then yes, that function-pointer indirection is going to be the bottleneck. If you mean that your assembly function is 1 μs per call and that's fast, given what it does, then I think it would be alright.

If you need to remove the function-pointer indirection and still run on Awkward Arrays, there are other things we can do, but they're more involved. Ping me in a GitHub Issue or Discussion on https://github.com/scikit-hep/awkward-1.0

Seems like this is a known bug in SQLModel: https://github.com/tiangolo/sqlmodel/issues/10

I have not used it myself but the creator of fastapi has made https://github.com/tiangolo/sqlmodel

I have already posted this question in Stack Overflow and GitHub and have been ignored in both 😢. You guys are my last hope.

I had a requirement to process and validate large payloads of JSON concurrently for a web service, initially I implemented it using jsonschema and fastjsonschema but I found the whole JSON Schema Specification to be confusing at times and on top of that wanted better performance. Albeit there are ways to compile/cache the schema, I wanted to move away from the schema specification so I wrote a validation library inspired by the design of tiangolo/sqlmodel (type hints) to solve this problem easier.

Github Repository

I would say as long as your current solution works and is easy to maintain keep it. If you want to switch I would recommend FastAPI, it is new(ish), but definitely old enough to have been tested and used in a large variety of production usecases. In your case it might be interesting to have a look at SQLModel (works with FastAPI, same author), especially if the API endpoints match closely to the database objects. https://github.com/tiangolo/sqlmodel

SQLModel.

Sounds similar to https://github.com/tiangolo/sqlmodel

There was a lot of hype and excitement around the release of SQLModel, a Pydantic + SQLAlchemy hybrid Model library with native integration for FastAPI. I pulled it out just now and there hasn't been any update beyond the initial anouncment on August 25th.