pgloader VS SQLAlchemy

No, it's difficult to read, and understand. It's a parenthesis circus, example -

https://github.com/dimitri/pgloader/blob/master/src/sources/...

Examples (for Common Lisp, so not citing Emacs): reddit v1, Google's ITA Software that powers airfare search engines (Kayak, Orbitz…), Postgres' pgloader (http://pgloader.io/), which was re-written from Python to Common Lisp, Opus Modus for music composition, the Maxima CAS, PTC 3D designer CAD software (used by big brands worldwide), Grammarly, Mirai, the 3D editor that designed Gollum's face, the ScoreCloud app that lets you whistle or play an instrument and get the music score,

but also the ACL2 theorem prover, used in the industry since the 90s, NASA's PVS provers and SPIKE scheduler used for Hubble and JWT, many companies in Quantum Computing, companies like SISCOG, who plans the transportation systems of european metropolis' underground since the 80s, Ravenpack who's into big-data analysis for financial services (they might be hiring), Keepit (https://www.keepit.com/), Pocket Change (Japan, https://www.pocket-change.jp/en/), the new Feetr in trading (https://feetr.io/, you can search HN), Airbus, Alstom, Planisware (https://planisware.com),

or also the open-source screenshotbot (https://screenshotbot.io), the Kandria game (https://kandria.com/),

and the companies in https://github.com/azzamsa/awesome-lisp-companies and on LispWorks and Allegro's Success Stories.

https://github.com/tamurashingo/reddit1.0/

http://opusmodus.com/

https://www.ptc.com/en/products/cad/3d-design

http://www.izware.com/mirai

https://apps.apple.com/us/app/scorecloud-express/id566535238

I worked on migrating our MySQL system to PostgreSQL using pgloader ( https://pgloader.io/ ).

There were some hiccups, things that needed clarification in documentation, and some additional processes that needed to be done outside of the system to get everything we need in place, it was a amazing help. Not sure the project would've been possible without it.

Data mapping from PostgreSQL to PostgreSQL as in the article isn't nearly as bad as going between systems. We took a full extended outage and didn't preload any data. There were many dry runs before hand and validation before hand, but the system wasn't so mission critical that we couldn't afford to shutoff the system for a couple of hours.

Initially, I started down the pgloader path, which seemed to be a common approach for the database conversion. However, using my M1-chip MacBook Pro led to some unexpected issues. Instead, I opted to use NMIG to convert MySQL to PostgreSQL. For more information, please check out the “Highlights From the Database Conversion” section below.

I just checked pgloader (which is mostly for heterogeneous database migrations) and it doesn't seem to have any explicit throttling features, either. However, you can set concurrency, number of workers, and batch size.

But do what makes you happy. The best PostgreSQL data loader in the world is written in Common Lisp. Now, CL is a fast and ANSI standardized language with multiple implementations, and was used to create Reddit, but it's hard to call it fashionable.

- [ScoreCloud](https://scorecloud.com/) - A web and mobile application to automatically create music notation from music performance or recordings. Built with LispWorks.

## DB tools

- [Pgloader](https://github.com/dimitri/pgloader/) - Migrate to PostgreSQL in a single command!. [PostgreSQL License]

I would warmly recommend https://pgloader.io but, unfortunately, the Oracle support is still in need of a sponsor :-)

I followed instruction here: https://github.com/dimitri/pgloader/blob/master/INSTALL.md, but can't get pass sbcl requirement.

https://pgloader.io/ There's been a great tool available for this for many years.

OK -

can we start considering binary files committed to a repo, even as data for tests, to be a huge red flag, and that the binary files themselves should instead be generated at testing time by source code that's stated as reviewable cleartext. This would make it much harder (though of course we can never really say "impossible") to embed a substantial payload in this way.

when binary files are part of a test suite, they are typically trying to illustrate some element of the program being tested, in this case a file that was incorrectly xz-encoded. Binary files like these weren't typed by hand, they will always ultimately come from something plaintext source.

Here's an example! My own SQLAlchemy repository has a few binary files in it! https://github.com/sqlalchemy/sqlalchemy/blob/main/test/bina... oh noes. Why are those files there? well in this case I just wanted to test that I can send large binary BLOBs into the database driver and I was lazy. This is actually pretty dumb, the two binary files here add 35K of useless crap to the source, and I could just as easily generate this binary data on the fly using a two liner that spits out random bytes. Anyone could see that two liner and know that it isn't embedding a malicious payload.

If I wanted to generate a poorly formed .xz file, I'd illustrate source code that generates random data, runs it through .xz, then applies "corruption" to it, like zeroing out the high bit of every byte. The process by which this occurs would be all reviewable in source code.

Besides, flama also provides support for SQL databases via SQLAlchemy, an SQL toolkit and Object Relational Mapper that gives application developers the full power and flexibility of SQL. Finally, flama also provides support for HTTP clients to perform requests via httpx, a next generation HTTP client for Python.

Alembic is a lightweight database migration tool for usage with SQLAlchemy. The term migration can be a little misleading, because in this context it doesn't mean to migrate to a different database in the sense of using a different version or a different type of database. In this context, migration refers to changes to the database schema: add a new column to a table, modify the type of an existing column, create a new index, etc..

To simplify this process, Cerbos developers have come up with adapters for popular Object-Relational Mapping (ORM) frameworks. You can check out for more details on the query plan repo - which also contains adapters for Prisma and SQLAlchemy - as well as a fully functioning application using Mongoose as its ORM.

That above pattern is one I've seen people do even recently, using the "select().c" attribute which from very early versions of SQLAlchemy is defined as "the columns from a subquery of the SELECT" ; this usage began raising deprecation warnings in 1.4 and is fully removed in 2.0 as it was a remnant of a much earlier version of SQLAlchemy. it will do exactly as you say, "make a subquery for each filter condition".

the moment you see SQLAlchemy doing something you see that seems "asinine", send an example to https://github.com/sqlalchemy/sqlalchemy/discussions and I will clarify what's going on, correct the usage so that the query you have is what you expect, and quite often we will add new warnings or documentation when we see people doing things we didn't anticipate.

The creator and lead maintainer of SQLAlchemy, one of the most popular and most used Python library for accessing databases (who doesn't?) gave a rather interesting response to PEP703.

If this doesn't ring any alarm bells I don't know what will.

> Basically for the moment the GIL-less idea would likely be burdensome for us and the fact that it's only an "option" seems to strongly imply major compatibility issues that we would not prefer.

https://github.com/sqlalchemy/sqlalchemy/discussions/10002#d...

Recently I discovered BigQuery public datasets - just over 200 datasets available for directly querying via SQL. I think this is a great thing! I can connect these direct to an analytics platform (we use Apache Superset which uses Python SQLAlchemy under the hood) for example and just start dashboarding.

When using python with sql databases like postgres or mariadb or SQLite you would use SQLAlchemy or another ORM of if you're feeling brave, you code it by hand. With ORMs you provide the address of your database and it connects for you, letting you use abstractions instead of writing all the SQL yourself (kind of analogous to using vlookups or index match instead of manually entering data).

Our tool we're going to be using for interfacing with the SQLite database is SQLAlchemy, a SQL toolkit that provides a unified API for various relational databases. If you installed FastAPI with pip install "fastapi[all]", SQLAlchemy is already part of your setup. but if you opted for FastAPI alone, you would need to install SQLAlchemy separately with pip install sqlalchemy.