imessage-exporter VS aifiles

I don't think this works with more recent iMessage features, it looks like it only queries the `text` column [0], but newer (i.e. post MacOS 13) require reading and parsing the attributed_body column [1].

[0]: https://github.com/niftycode/imessage_reader/blob/master/ime...

[1]: https://github.com/ReagentX/imessage-exporter/blob/2dc3d034b...

Hey, this sounds like an interesting problem. I am always looking for edge cases to test, if you have time would you mind checking if https://github.com/ReagentX/imessage-exporter works for you and if it crashes in that spot?

I wrote a tool for this: https://github.com/ReagentX/imessage-exporter

If you are talking about data missing from the `text` column, for some reason it disappears after you read a message. The content is stored in a binary blob in a different column, which I parse like this: https://github.com/ReagentX/imessage-exporter/blob/c73bc4d66...

As someone who has moved between macOS+iOS and Windows/Linux+Android several times, this is doable. For technical people, it's just annoying. For non-technical people, this probably needs written into a more formal set of steps. If you have the need, one can setup your world to work on both systems transparently, but that takes more work.

Caveats: iOS messages can be kept, but they'll be in files, not your new message app. Photo edits will be lost unless you take extra steps.

Messages: if you are comfortable with it, use imessage-exporter (https://github.com/ReagentX/imessage-exporter) on your Mac to export your messages to disk. Copy to new machine. Validate you got what you wanted! Can also be used to decrease iCloud usage by backing up messages and deleting the originals from Apple Messages.

Photos: three ways. 1) open up the macOS photos app, select all photos, and export them. This will make any JPEG photos much larger than they originally were due to ridiculous default quality settings. 2) If you have access to a Windows machine, install iCloud for Windows, let the photos sync, and copy them to a new directory. 3) Use iCloud's web UI to download all the photos on the new machine.

Mail: pick a new provider. Several ways. 1) Add the new provider account to macOS mail. Copy and paste your emails/folders between accounts. 2) Create an app-specific password for iCloud, use the provider's migration facility (most major players support this and it will move your contacts and calendars).

Calendars: if you are sharing calendars with iOS users or will keep some Apple devices, keep iCloud as your primary calendar system. Use DavX5 (https://www.davx5.com/) on Android to setup a two-way mirror between your Android calendar app. Your email provider may provide calendar mirroring (Fastmail does, for one). If you aren't sharing / using your mail provider, export your calendars to ICS files from Apple Calendar and import into your new calendar app.

Contacts: if you will continue using Apple products, keep iCloud as your primary contacts system and use DavX5. Otherwise, open macOS Contacts, select all, and export to a VCF file. Import this into your new Contacts app.

Documents: copy to a backup drive from the machine, download them from iCloud on the web, use iCloud for Windows for the initial sync. Whatever suits you. If you are using Apple's office apps, be sure to load and save as a more universal format.

Since you have a Mac you can use this program: https://github.com/ReagentX/imessage-exporter

So yes, those 10TB archives may end up being 5TB if someone spent the time to really comb over, understand, make good decisions, and organize that data. But I have not yet seen anything that can scratch that surface yet, other than perhaps https://github.com/jjuliano/aifiles - but I won't use it until it's local only and has guarantees not to destroy data without explicit permission. An overlay filesystem that shows compression/deduplication with LLM capability like aifiles is probably the best option here.

However, I wouldn't imagine that most people's life data is less than 2TB even with all of this - it's mostly imposed as an artificial constraint by these companies.

I imagine things like this are underway: https://github.com/jjuliano/aifiles

Honestly I think it's actually a pretty fantastic development if that's the direction. I don't use Dropbox much, nor have I used aifiles (due to sending all of your data to OpenAI) - but the idea of being able to not have to manually and tediously look over terabytes of files to completely reorganize all of your files into a better directory hierarchy, and tag each files with meaningful labels, etc sounds phenomenal.

Obviously there are some implementation details for this to be not awful - for example: 1) only local models for local data, 2) making the changes on e.g. ZFS (to allow rollback) or as some type of optional 'overlay' view to switch back and forth from the original to ai-organized, etc. and 3) having thresholds and logic for what may be considered 'duplicates' to be removed, and how to better compress data

As for the de-duplication and processing: this could be very good for dropbox in that, e.g. if a person wants to completely re-encode all of their image files or video files with AV1, the resulting data could be cut in half or more - which saves Dropbox storage space. After which, neural perceptual hashing could be done on all of the files and a threshold of similarity could do de-duplication on a perceptual basis (for example, keep the bigger size file that is 99% similar to a 2x downsized version, and re-encode it). User preference to keep things like tiff files completely intact or any other lossless encoding of their choosing could be good options as well

There's definitely a strange disparity between the computation cost for deploying a decent model to do this compared to the storage cost - but if a (perhaps even non-LLM!) small model is created to be able to plow through data at fast rates could be deployed it may make sense.

Or perhaps the type of semantic compression that LLMs do are of interest for making a new type of lossy compression algorithm of which Dropbox is interested.

aifiles: A CLI that organize and manage your files using AI

Output from the language model is also being injected into a script that is then executed: https://github.com/jjuliano/aifiles/blob/ef529fd6281eaf8d373...

He argued below that he is not vulnerable to indirect prompt injection attacks (https://github.com/greshake/llm-security), but I think he is wrong.