chardet VS MySQLTuner-perl

chardet is a library for decoding characters, once installed you can use the following to determine encoding:

The craziest issue I had was I couldn't predict what char encoding the text in my database was in. Most users entered Windows-1252, some text blobs were UTF-16, others were European character sets, and some were UTF-8. Some were Japanese SHIFT_JIS. Don't ask me how any of this happened. I retrospect, I should have dumped all the tables from MySQL and used the excellent PyPy Chardet [1] library to see what I was dealing with, do the conversions and then re-import the data. But then someone could copy UTF-16 from a Windows document and paste it in, so you have to convert going in to the database.

You have set Apache to UTF-8, PHP to UTF-8, MySQL to UTF-8, and the MySQL driver you are using to UTF-8. It's not clear how these setting interact. Are there silent conversions happening or do you always have to detect the encoding on data coming from the server? HTML pages have a character encoding specifier, but the BOM at the start of the file takes precedence (I think.) I got it to work by always detecting encoding for any text coming from the database and using iconv, but this turned out to be really slow and unreliable. It was truly the biggest mess by an order of magnitude than any other programming problem I faced in my career.

Would not attempt again.

[1] https://github.com/chardet/chardet

I found there is a https://github.com/chardet/chardet python library, which can be ported to Common Lisp.

Then use chardet to determine the encoding from the content

The library that Requests uses for content encoding detection has for the past 10 years been chardet which is licensed LGPL-2.1.

What helped me for one server was dedicating more of the memory to the database storage engine. In my case it was InnoDB buffer pool tweaks. If you’re running MySQL then using MySQLTuner (https://github.com/major/MySQLTuner-perl) may help but this all requires command line work so I would suggest this only being done if you’re experienced with this or getting someone experienced to help. I’ve also seen but not yet experienced that a managed database solution or a separate database server may help.

If you set the innodb_buffer_pool_size to roughly 80% of the total available memory everything gets cached and run from memory. Have you tried this? And have you tried just running mysqltuner and see what that suggests? https://github.com/major/MySQLTuner-perl

One of the biggest and best tools is MySQL Tuner and this will be good to run as it will give you specifics based on your hardware. You can also try SolarWinds. https://github.com/major/MySQLTuner-perl

Are the DBA's on this thread committing code to the MySQL [1] and Postgres [2] tuner scripts that give new DBA's all your learned advise and battle hardening experience? Or at least a repo with the same objectives?

[1] - https://github.com/major/MySQLTuner-perl

[2] - https://github.com/jfcoz/postgresqltuner

You could try to manage this query_cache_size. If you use phpmyadmin there is a tool : phpMyAdmin Advisor, it will give you advices ! Make sure your tables are innodb, not myisam. And this tool also https://github.com/major/MySQLTuner-perl I Always change one parameters at a time. And step by step to avoid unatempted effects.

You can also try out MySQL Tuner to identify some opportunities for configuration improvements. Just make sure that you thoroughly understand the settings and the effects of any changes that you might consider making since you are in a production environment.

Not a DBA, but I've had lots of success with MySQL Tuner (https://github.com/major/MySQLTuner-perl)