chardet VS docker-postgres-upgrade

chardet is a library for decoding characters, once installed you can use the following to determine encoding:

The craziest issue I had was I couldn't predict what char encoding the text in my database was in. Most users entered Windows-1252, some text blobs were UTF-16, others were European character sets, and some were UTF-8. Some were Japanese SHIFT_JIS. Don't ask me how any of this happened. I retrospect, I should have dumped all the tables from MySQL and used the excellent PyPy Chardet [1] library to see what I was dealing with, do the conversions and then re-import the data. But then someone could copy UTF-16 from a Windows document and paste it in, so you have to convert going in to the database.

You have set Apache to UTF-8, PHP to UTF-8, MySQL to UTF-8, and the MySQL driver you are using to UTF-8. It's not clear how these setting interact. Are there silent conversions happening or do you always have to detect the encoding on data coming from the server? HTML pages have a character encoding specifier, but the BOM at the start of the file takes precedence (I think.) I got it to work by always detecting encoding for any text coming from the database and using iconv, but this turned out to be really slow and unreliable. It was truly the biggest mess by an order of magnitude than any other programming problem I faced in my career.

Would not attempt again.

[1] https://github.com/chardet/chardet

I found there is a https://github.com/chardet/chardet python library, which can be ported to Common Lisp.

Then use chardet to determine the encoding from the content

The library that Requests uses for content encoding detection has for the past 10 years been chardet which is licensed LGPL-2.1.

Ah, looks like psql seems to have switched to actually using minor and major versions properly during the last two years - I remember the dance from the 9.x versions and postgres-upgrade [1].

In any case it's way more straightforward with mysql.

[1] https://hub.docker.com/r/tianon/postgres-upgrade

Related: https://github.com/tianon/docker-postgres-upgrade

Also don't forget to pg_dump before the upgrade, as the binaries for previous Postgres version might not exist anymore after the upgrade... Otherwise, https://hub.docker.com/r/tianon/postgres-upgrade/ might be helpful (haven't used it though).

> When you say "complex ride" what does that mean?

On managed postgresql services like RDS it's easy, yes, because Amazon does all of the nasty work for you and has already seen all the kinks you may run into.

If you are on your own and using Docker, you essentially need a third-party image that has the old and new binaries (https://github.com/tianon/docker-postgres-upgrade).

If you are on your own and are using distribution packages, it's more complicated (https://blog.samuel.domains/blog/tutorials/from-stretch-to-b...).

If you are on your own and use postgres from source, well... have fun, there's a reason why I prefer using distribution-maintained packages when possible.