humility
xsv
Our great sponsors
humility | xsv | |
---|---|---|
6 | 64 | |
512 | 10,080 | |
2.5% | - | |
8.2 | 0.0 | |
7 days ago | 2 months ago | |
Rust | Rust | |
Mozilla Public License 2.0 | The Unlicense |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
humility
-
Barracuda Urges Replacing – Not Patching – Its Email Security Gateways
A lot of questions in there! Taking these in order:
1. We aren't making standalone servers: the Oxide compute sled comes in the Oxide rack. So are not (and do not intend to be) a drop in replacement for extant rack mounted servers.
2. We have taken a fundamentally different approach to firmware, with a true root of trust that can attest to the service processor -- which can turn attest to the system software. This prompts a lot of questions (e.g., who attests to the root of trust?), and there is a LOT to say about this; look for us to talk a lot more about this
3. In stark contrast (sadly) to nearly everyone else in the server space, the firmware we are developing is entirely open source. More details on that can be found in Cliff Biffle's 2021 OSFC talk and the Hubris and Humility repos.[0][1][2]
4. Definitely not vaporware! We are in the process of shipping to our first customers; you can follow our progress in our Oxide and Friends podcast.[3]
[0] https://www.osfc.io/2021/talks/on-hubris-and-humility-develo...
[1] https://github.com/oxidecomputer/hubris
[2] https://github.com/oxidecomputer/humility
[3] https://oxide-and-friends.transistor.fm/
- Do you use Rust in your professional career?
-
What's the project you're currently working on at your company as a Rust developer?
It's a mix of embedded work and improving the system's tooling (faster builds, debugger support, etc)
-
Oxide on My Wrist: Hubris on PineTime was the best worst idea
Other folks have mentioned this, but it's important to understand the limitations of Rust with respect to safety. In particular: every stack operation is -- at some level -- an unsafe operation as it operates without a bounds check. This isn't Rust's fault per se; non-segmented architectures don't have an architecturally defined way to know the stack base. As a result, even an entirely safe Rust program can make an illegal access to memory that results in fatal program failure. That, of course, assumes memory protection; if you don't have memory protection (or, like many embedded operating systems, you don't make use of it), stack overflows will plow into adjacent memory.
But wait, it gets worse: stack overflows are often not due to infinite stack consumption (e.g., recursion) but rather simply going deep on an unusual code path. If stack consumption just goes slightly beyond the base of the stack and there is no memory protection, this is corrupt-and-run -- and you are left debugging a problem that looks every bit like a gnarly data race in an unsafe programming language. And this problem becomes especially acute when memory is scarce: you really don't want a tiny embedded system to be dedicating a bunch of its memory to stack space that will never ("never") be used, so you make the stacks as tight as possible -- making stack overflows in fact much more likely.
Indeed, even with the MPU, these problems were acute in the development of Hubris: we originally put the stack at the top of a task's data space, and its data at the bottom -- and we found that tasks that only slightly exceeded their stack (rather than running all of the way through its data and into the protection boundary) were corrupting themselves with difficult-to-debug failures. We flipped the order to assure that every stack overflow hit the protection boundary[0], which required us to be much more intentional about the stack versus data split -- but had the added benefit of allowing us to add debugging support for it.[1]
Stack overflows are still pesky (and still a leading cause of task death!), but without the MPU, each one of these stack overflows would be data corruption -- answering for us viscerally what we "need the MPU for."
[0] https://github.com/oxidecomputer/hubris/commit/d75e832931f67...
[1] https://github.com/oxidecomputer/humility#humility-stackmarg...
-
Writing embedded firmware using Rust
In addition to Cliff's talk/blog -- which are absolutely outstanding -- I would recommend listening to the Twitter Space we did on Hubris and Humility last week.[0] It was a really fun conversation, and it also serves as a bit of a B-side for the talk in that it goes into some of the subtler details that we feel are important, but didn't quite rise to the level of the presentation. And of course, be sure to check out the source itself![1][2]
[0] https://www.youtube.com/watch?v=cypmufnPfLw
[1] https://github.com/oxidecomputer/hubris
[2] https://github.com/oxidecomputer/humility
-
Hubris - OS for embedded computer systems
Humility (the debugger)
xsv
-
Show HN: TextQuery – Query and Visualize Your CSV Data in Minutes
I realize it's not really that comparable since these tools don't support SQL, but a more fully functioned CLI tool is - https://github.com/BurntSushi/xsv
They are both fairly good
- Qsv: Efficient CSV CLI Toolkit
-
Joining CSV Data Without SQL: An IP Geolocation Use Case
I have done some similar, simpler data wrangling with xsv (https://github.com/BurntSushi/xsv) and jq. It could process my 800M rows in a couple of minutes (plus the time to read it out from the database =)
-
Qsv: CSVs sliced, diced and analyzed (fork of xsv)
xsv, which seems to be why qsv was created.
[1] https://github.com/BurntSushi/xsv/issues/267
-
I wrote this iCalendar (.ics) command-line utility to turn common calendar exports into more broadly compatible CSV files.
CSV utilities (still haven't pick a favorite one...): https://github.com/harelba/q https://github.com/BurntSushi/xsv https://github.com/wireservice/csvkit https://github.com/johnkerl/miller
- Icsp – Command-line iCalendar (.ics) to CSV parser
-
ripgrep is faster than {grep, ag, git grep, ucg, pt, sift}
$ git remote -v origin [email protected]:rust-lang/rust (fetch) origin [email protected]:rust-lang/rust (push) $ git rev-parse HEAD 3b0d4813ab461ec81eab8980bb884691c97c5a35 $ time grep -ri burntsushi ./ ./src/tools/cargotest/main.rs: repo: "https://github.com/BurntSushi/ripgrep", ./src/tools/cargotest/main.rs: repo: "https://github.com/BurntSushi/xsv", grep: ./target/debug/incremental/cargotest-2dvu4f2km9e91/s-gactj3ma2j-1b10l4z-2l60ur55ixe6n/query-cache.bin: binary file matches grep: ./target/debug/incremental/cargotest-38cpmhhbdgdyq/s-gactj3luwq-1o12vgp-t61hd8qdyp7t/query-cache.bin: binary file matches grep: ./target/debug/incremental/cargotest-17632op6djxne/s-gawuq5468i-1h69nfw-4gm0s8yhhiun/query-cache.bin: binary file matches grep: ./target/debug/incremental/cargotest-2trm4kt5yom3r/s-gawuq53qqg-bjiezj-lo0gha8ign8w/query-cache.bin: binary file matches grep: ./target/debug/deps/libregex_automata-c74a6d9fd0abd77b.rmeta: binary file matches grep: ./target/debug/deps/libsame_file-a0e0363a2985455d.rlib: binary file matches grep: ./target/debug/deps/libsame_file-a0e0363a2985455d.rmeta: binary file matches grep: ./target/debug/deps/libsame_file-7251d8d3586a319b.rmeta: binary file matches grep: ./build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaho_corasick-999a08e2b700420d.rlib: binary file matches grep: ./build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libregex_automata-0d168be5d25b3ac5.rlib: binary file matches grep: ./build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/deps/libregex_automata-7d6bec0156f15da1.rlib: binary file matches grep: ./build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/deps/libregex_automata-7d6bec0156f15da1.rmeta: binary file matches grep: ./build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/deps/libaho_corasick-07dee4514b87d99b.rmeta: binary file matches grep: ./build/x86_64-unknown-linux-gnu/stage0-tools/x86_64-unknown-linux-gnu/release/deps/libaho_corasick-07dee4514b87d99b.rlib: binary file matches grep: ./build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libaho_corasick-999a08e2b700420d.rlib: binary file matches grep: ./build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libaho_corasick-999a08e2b700420d.rmeta: binary file matches grep: ./build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libregex_automata-0d168be5d25b3ac5.rlib: binary file matches grep: ./build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/libregex_automata-0d168be5d25b3ac5.rmeta: binary file matches grep: ./build/bootstrap/debug/deps/libaho_corasick-992e1ba08ef83436.rmeta: binary file matches grep: ./build/bootstrap/debug/deps/libignore-54d41239d2761852.rmeta: binary file matches grep: ./build/bootstrap/debug/deps/libsame_file-9a5e3ddd89cfe599.rlib: binary file matches grep: ./build/bootstrap/debug/deps/libregex_automata-8e700951c9869a66.rlib: binary file matches grep: ./build/bootstrap/debug/deps/libignore-54d41239d2761852.rlib: binary file matches grep: ./build/bootstrap/debug/deps/libaho_corasick-992e1ba08ef83436.rlib: binary file matches grep: ./build/bootstrap/debug/deps/libregex_automata-8e700951c9869a66.rmeta: binary file matches grep: ./build/bootstrap/debug/deps/libsame_file-9a5e3ddd89cfe599.rmeta: binary file matches real 16.683 user 15.793 sys 0.878 maxmem 8 MB faults 0
-
Any Linux admins willing to try Pygrep?
Unrelated, are you the same burntsushi that wrote xsv?
-
Analyzing multi-gigabyte JSON files locally
If it could be tabular in nature, maybe convert to sqlite3 so you can make use of indexing, or CSV to make use of high-performance tools like xsv or zsv (the latter of which I'm an author).
https://github.com/BurntSushi/xsv
https://github.com/liquidaty/zsv/blob/main/docs/csv_json_sql...
-
What monitoring tool do you use or recommend?
Oh and there's rad cli shit out there for CSV files too, like xsv
What are some alternatives?
tock - A secure embedded operating system for microcontrollers
csvtk - A cross-platform, efficient and practical CSV/TSV toolkit in Golang
esp32-hal - A hardware abstraction layer for the esp32 written in Rust.
miller - Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
hubris - A lightweight, memory-protected, message-passing kernel for deeply embedded systems.
ripgrep - ripgrep recursively searches directories for a regex pattern while respecting your gitignore
fathom - 🚧 (Alpha stage software) A declarative data definition language for formally specifying binary data formats. 🚧
Servo - Servo, the embeddable, independent, memory-safe, modular, parallel web rendering engine
InfiniTime - Firmware for Pinetime smartwatch written in C++ and based on FreeRTOS
Fractalide - Reusable Reproducible Composable Software
esp32 - Peripheral access crate for the ESP32
svgcleaner - svgcleaner could help you to clean up your SVG files from the unnecessary data.