spyql
cutter
spyql | cutter | |
---|---|---|
23 | 39 | |
902 | 15,136 | |
- | 2.0% | |
0.0 | 8.2 | |
over 1 year ago | 2 days ago | |
Jupyter Notebook | C++ | |
MIT License | GNU General Public License v3.0 only |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
spyql
-
Fq: Jq for Binary Formats
I prefer a SQL-like format. It’s not as complete but it cover most of the day-to-day use cases. Take a look at https://github.com/dcmoura/spyql (I am the author). Congrats on fq!
-
Command-line data analytics made easy with SPyQL
SPyQL documentation: spyql.readthedocs.io
-
This Week In Python
spyql – Query data on the command line with SQL-like SELECTs powered by Python expressions
- Command-line data analytics made easy
-
Jc – JSONifies the output of many CLI tools
This is great!
I am the author of SPyQL [1]. Combining JC with SPyQL you can easily query the json output and run python commands on top of it from the command-line :-) You can do aggregations and so forth in a much simpler and intuitive way than with jq.
I just wrote a blogpost [2] that illustrates it. It is more focused on CSV, but the commands would be the same if you were working with JSON.
[1] https://github.com/dcmoura/spyql
- The fastest command-line tools for querying large JSON datasets
-
Working with more than 10gb csv
You can import the data into a PostgreSQL/MySQL/SQLite/... database and then query the database. However, even with the right choice of indexes, it might take a while to run queries on a table with hundreds of millions of records. You can easily import your data to these databases with SpyQL: $ spyql "SELECT * FROM csv TO sql(table=my_table_name) | sqlite3 my.db" (you would need to create the table my_table_name before running the command).
-
ClickHouse Cloud is now in Public Beta
https://github.com/dcmoura/spyql/blob/master/notebooks/json_...
And ClickHouse looks like a normal relational database - there is no need for multiple components for different tiers (like in Druid), no need for manual partitioning into "daily", "hourly" tables (like you do in Spark and Bigquery), no need for lambda architecture... It's refreshing how something can be both simple and fast.
- A SQLite extension for reading large files line-by-line
-
I want to convert a large JSON file into Tabular Format.
I thought this library was pretty nifty for json. It's also relatively fast compared to most json parsers: https://github.com/dcmoura/spyql
cutter
-
The Hiew Hex Editor
Everything Hiew can do, Rizin[1] can do too, and is completely free and open source[2] under LGPL3 license. Moreover, it supports more architectures, platforms, and file formats, as well as GUI in Qt - Cutter[3][4]. If something is missing in Rizin but presented in Hiew, please let us know by opening the issue with details.
[1] https://rizin.re
[2] https://github.com/rizinorg/rizin
[3] https://cutter.re
[4] https://github.com/rizinorg/cutter
-
If you're interested in eye-tracking, I'm interested in funding you
Okay, so, your comment about a "Dasher + Guitar Hero music theory/improvisation practice program" just sent me down a huge rabbit hole...
Well, rabbit hole(s) plural, I guess, most not directly related. :D
Largely because I made the "mistake" of looking at your HN profile & discovering you're also in NZ & we seem to have somewhat overlapping interests (and an affinity for "bacon" in account names, apparently), so, some thoughts[0]... :)
# Topic 1: Nissan Leaf VSP hacking
After reading your recent posts (https://ianrrees.github.io//2023/07/03/vsp-hacking.html & https://ianrrees.github.io//2023/08/05/voltage-glitch-inject...) on this topic & noting your remark about wanting to try reverse engineering a firmware image, I found the following thesis PDF (via a brief google search for `"reverse engineer" "firmware" "Renesas"`):
* "AUTOMOTIVE FIRMWARE EXTRACTION AND ANALYSIS TECHNIQUES" by Jan Van den Herrewegen https://etheses.bham.ac.uk/id/eprint/11516/1/VandenHerrewege...
Not really what I was anticipating finding but seems relevant to your interests--I don't think it was already in your resource list.
While the thesis addresses the Renesas 78K0 rather than the Renesas 78K0R, from a brief look at the "Flash Protection" PDF Application Note in your resource list it seems there's a large overlap.
Perhaps most significantly the author presents "novel methods" that combine bootloader binary analysis with constraint-based power glitching in an effort to improve on the results described in "Shaping the Glitch".
While I haven't read the entire 186 pages :D they theorize that using their approach extracting 8kB firmware might only take ~10 hours.
And, most helpfully, they even published their source code under the GPL here: https://github.com/janvdherrewegen/bootl-attacks
So, an interesting adjacent read even if it turns out not to be directly applicable to your situation.
Given I have an interest in & a little experience with firmware reversing my original thought was to maybe provide some hopefully helpful references that more generically related to firmware reversing but more specific is good too, I guess. :)
In terms of reverse engineering tooling, I've used Rizin/Cutter/radare2 previously: https://rizin.re https://cutter.re
On the CAN tooling/info front, you might be interested in taking a look at my "Adequate CAN" list which I originally wrote-up for a client a couple years ago: https://gitlab.com/RancidBacon/adequate-can
Some other probably outdated reverse engineering tooling links of mine: https://web.archive.org/web/20200119074540/http://www.labrad...
In terms of how to approach RE, other than just "getting started & digging in" & learning by doing, I've sometimes found it informative to read other people's firmware reverse engineering write-ups to learn about potentially useful approaches/tools.
Anyway, hopefully some of this is helpful!
[0] I have a tendency to be a little... "verbose" and/or "thorough" (depending on one's POV :) ) so I'll probably split this over a couple of comments, in case I run out of steam while writing and for topic separation.
-
Veles – A new age tool for binary analysis
In Cutter[1][2] we have an idea to implement thes same feature[3] as a plugin, but our priorities lie elsewhere die to the lack of enough hands. Contributions are welcome.
[1] https://cutter.re
[2] https://github.com/rizinorg/cutter
[3] https://github.com/rizinorg/cutter-plugins/issues/3
- Debugger Ghidra Class
-
Fq: Jq for Binary Formats
For this kind of task, using low-level debugger tools is probably better. Rizin[1][2]/Cutter[3][4] could help. We also have GSoC participant this year who works hard on improving debuginfo and debugging support[5]. I personally also like Binary Ninja, they recently made their debugger stable enough[6].
[1] https://rizin.re/
[2] https://github.com/rizinorg/rizin
[3] https://cutter.re/
[4] https://github.com/rizinorg/cutter
[5] https://rizin.re/posts/gsoc-2023-announcement/
[5] https://binary.ninja/2023/05/03/3.4-finally-freed.html#debug...
- Cutter (Reverse Engineering Tool) v2.2.1
-
What is this?
Something like https://cutter.re/ or https://www.nirsoft.net/utils/dll_export_viewer.html Could possibly give you some insight. I guess the question though is, what are you trying to do with it?
- Cutter Release 2.2.0
-
Dis This: Disassemble Python code online
Rizin[1] (and therefore Cutter[2]) supports interactive disassembly and analysis (but not decompilation) of the Python bytecode[3][4]. Apart from that it also supports Java and Lua bytecode for different versions.
[1] https://rizin.re
[2] https://cutter.re
[3] https://github.com/rizinorg/rizin/tree/dev/librz/asm/arch/py...
[4] https://github.com/rizinorg/rizin/blob/dev/librz/analysis/p/...
-
Stuff like this is why everyone uses scripting languages these days, since the main value prop of high-level languages is their ability to reliably print backtraces.
cutter is quite nice though
What are some alternatives?
prql - PRQL is a modern language for transforming data — a simple, powerful, pipelined SQL replacement
ghidra - Ghidra is a software reverse engineering (SRE) framework
malloy - Malloy is an experimental language for describing data relationships and transformations.
rz-ghidra - Deep ghidra decompiler and sleigh disassembler integration for rizin
tresql - Shorthand SQL/JDBC wrapper language, providing nested results as JSON and more
rizin - UNIX-like reverse engineering framework and command-line toolset.
Preql - An interpreted relational query language that compiles to SQL.
r2ghidra - Native Ghidra Decompiler for r2
prosto - Prosto is a data processing toolkit radically changing how data is processed by heavily relying on functions and operations with functions - an alternative to map-reduce and join-groupby
efiSeek - Ghidra analyzer for UEFI firmware.
pxi - 🧚 pxi (pixie) is a small, fast, and magical command-line data processor similar to jq, mlr, and awk.
AppImageLauncher - Helper application for Linux distributions serving as a kind of "entry point" for running and integrating AppImages