spyql VS cutter

I prefer a SQL-like format. It’s not as complete but it cover most of the day-to-day use cases. Take a look at https://github.com/dcmoura/spyql (I am the author). Congrats on fq!

SPyQL documentation: spyql.readthedocs.io

spyql – Query data on the command line with SQL-like SELECTs powered by Python expressions

This is great!

I am the author of SPyQL [1]. Combining JC with SPyQL you can easily query the json output and run python commands on top of it from the command-line :-) You can do aggregations and so forth in a much simpler and intuitive way than with jq.

I just wrote a blogpost [2] that illustrates it. It is more focused on CSV, but the commands would be the same if you were working with JSON.

[1] https://github.com/dcmoura/spyql

You can import the data into a PostgreSQL/MySQL/SQLite/... database and then query the database. However, even with the right choice of indexes, it might take a while to run queries on a table with hundreds of millions of records. You can easily import your data to these databases with SpyQL: $ spyql "SELECT * FROM csv TO sql(table=my_table_name) | sqlite3 my.db" (you would need to create the table my_table_name before running the command).

https://github.com/dcmoura/spyql/blob/master/notebooks/json_...

And ClickHouse looks like a normal relational database - there is no need for multiple components for different tiers (like in Druid), no need for manual partitioning into "daily", "hourly" tables (like you do in Spark and Bigquery), no need for lambda architecture... It's refreshing how something can be both simple and fast.

I thought this library was pretty nifty for json. It's also relatively fast compared to most json parsers: https://github.com/dcmoura/spyql

Everything Hiew can do, Rizin[1] can do too, and is completely free and open source[2] under LGPL3 license. Moreover, it supports more architectures, platforms, and file formats, as well as GUI in Qt - Cutter[3][4]. If something is missing in Rizin but presented in Hiew, please let us know by opening the issue with details.

[1] https://rizin.re

[2] https://github.com/rizinorg/rizin

[3] https://cutter.re

[4] https://github.com/rizinorg/cutter

Okay, so, your comment about a "Dasher + Guitar Hero music theory/improvisation practice program" just sent me down a huge rabbit hole...

Well, rabbit hole(s) plural, I guess, most not directly related. :D

Largely because I made the "mistake" of looking at your HN profile & discovering you're also in NZ & we seem to have somewhat overlapping interests (and an affinity for "bacon" in account names, apparently), so, some thoughts[0]... :)

# Topic 1: Nissan Leaf VSP hacking

After reading your recent posts (https://ianrrees.github.io//2023/07/03/vsp-hacking.html & https://ianrrees.github.io//2023/08/05/voltage-glitch-inject...) on this topic & noting your remark about wanting to try reverse engineering a firmware image, I found the following thesis PDF (via a brief google search for `"reverse engineer" "firmware" "Renesas"`):

* "AUTOMOTIVE FIRMWARE EXTRACTION AND ANALYSIS TECHNIQUES" by Jan Van den Herrewegen https://etheses.bham.ac.uk/id/eprint/11516/1/VandenHerrewege...

Not really what I was anticipating finding but seems relevant to your interests--I don't think it was already in your resource list.

While the thesis addresses the Renesas 78K0 rather than the Renesas 78K0R, from a brief look at the "Flash Protection" PDF Application Note in your resource list it seems there's a large overlap.

Perhaps most significantly the author presents "novel methods" that combine bootloader binary analysis with constraint-based power glitching in an effort to improve on the results described in "Shaping the Glitch".

While I haven't read the entire 186 pages :D they theorize that using their approach extracting 8kB firmware might only take ~10 hours.

And, most helpfully, they even published their source code under the GPL here: https://github.com/janvdherrewegen/bootl-attacks

So, an interesting adjacent read even if it turns out not to be directly applicable to your situation.

Given I have an interest in & a little experience with firmware reversing my original thought was to maybe provide some hopefully helpful references that more generically related to firmware reversing but more specific is good too, I guess. :)

In terms of reverse engineering tooling, I've used Rizin/Cutter/radare2 previously: https://rizin.re https://cutter.re

On the CAN tooling/info front, you might be interested in taking a look at my "Adequate CAN" list which I originally wrote-up for a client a couple years ago: https://gitlab.com/RancidBacon/adequate-can

Some other probably outdated reverse engineering tooling links of mine: https://web.archive.org/web/20200119074540/http://www.labrad...

In terms of how to approach RE, other than just "getting started & digging in" & learning by doing, I've sometimes found it informative to read other people's firmware reverse engineering write-ups to learn about potentially useful approaches/tools.

Anyway, hopefully some of this is helpful!

[0] I have a tendency to be a little... "verbose" and/or "thorough" (depending on one's POV :) ) so I'll probably split this over a couple of comments, in case I run out of steam while writing and for topic separation.

In Cutter[1][2] we have an idea to implement thes same feature[3] as a plugin, but our priorities lie elsewhere die to the lack of enough hands. Contributions are welcome.

[1] https://cutter.re

[2] https://github.com/rizinorg/cutter

[3] https://github.com/rizinorg/cutter-plugins/issues/3

For this kind of task, using low-level debugger tools is probably better. Rizin[1][2]/Cutter[3][4] could help. We also have GSoC participant this year who works hard on improving debuginfo and debugging support[5]. I personally also like Binary Ninja, they recently made their debugger stable enough[6].

[1] https://rizin.re/

[2] https://github.com/rizinorg/rizin

[3] https://cutter.re/

[4] https://github.com/rizinorg/cutter

[5] https://rizin.re/posts/gsoc-2023-announcement/

[5] https://binary.ninja/2023/05/03/3.4-finally-freed.html#debug...

Something like https://cutter.re/ or https://www.nirsoft.net/utils/dll_export_viewer.html Could possibly give you some insight. I guess the question though is, what are you trying to do with it?

Rizin[1] (and therefore Cutter[2]) supports interactive disassembly and analysis (but not decompilation) of the Python bytecode[3][4]. Apart from that it also supports Java and Lua bytecode for different versions.

[1] https://rizin.re

[2] https://cutter.re

[3] https://github.com/rizinorg/rizin/tree/dev/librz/asm/arch/py...

[4] https://github.com/rizinorg/rizin/blob/dev/librz/analysis/p/...

cutter is quite nice though