xv6-public
pgloader
xv6-public | pgloader | |
---|---|---|
25 | 31 | |
7,408 | 5,089 | |
1.3% | - | |
0.0 | 3.0 | |
5 days ago | about 1 month ago | |
C | Common Lisp | |
GNU General Public License v3.0 or later | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
xv6-public
-
Challenging projects every programmer should try
+1 for mini operating system.
Us, application developers, rely on many OS features: memory management, filesystem, etc. I'm sure eventually we'll ask "how such things are done behind the scene?"
That's why I tinker with xv6 (https://github.com/mit-pdos/xv6-public) during sparetime. Learning various process scheduling algorithms from textbook is a thing. Implementing it is another thing. I learn a lot. And it's definitely fun, even though there's almost zero chance the knowledge gained is relevant for my job (I'm a mobile app dev).
-
xv6 compile error
Recently I compiled xv6 using gcc 7.5.0 on Ubuntu 18 , everything is ok. But when I try to compile it using gcc 13.2.1 on latest Arch, it's failed: result
-
How could the early Unix OS comprise so few lines of code?
https://github.com/mit-pdos/xv6-public has under 10,000 lines of C and assembly including some user space programs.
-
The rxv64 Operating System: MIT's xv6, in Rust, for SMP x86_64 machines
xv6 was originally written for 32-bit x86; the RISC-V port is a relatively recent development. See e.g. https://github.com/mit-pdos/xv6-public for some of the earlier history.
rxv64 was written for a specific purpose: we had to ramp up professional engineers on both 64-bit x86_64 and kernel development in Rust; we were pointing them to the MIT materials, which at the time still focused on x86, but they were getting tripped up 32-bit-isms and the original PC peripherals (e.g., accessing the IDE disk via programmed IO). Interestingly, the non sequitur about C++ aside, porting to Rust exposed several bugs or omissions in the C original; fixes were contributed back to MIT and applied to the original (and survived into the RISC-V port).
Oh, by the way, the use of the term "SMP" predates Intel's usage by decades.
-
Some were meant for C [pdf]
I'd define an arena as the pattern where the arena itself owns N objects. So you free the arena to free all objects.
My first job was at EA working on console games (PS2, GameCube, XBox, no OS or virtual memory on any of them), and while at the time I was too junior to touch the memory allocators themselves, we were definitely not malloc-ing and freeing all the time.
It was more like you load data for the level in one stage, which creates a ton of data structures, and then you enter a loop to draw every frame quickly. There were many global variables.
---
Wikipedia calls it a region, zone, arena, area, or memory context, and that seems about right:
https://en.wikipedia.org/wiki/Region-based_memory_management
It describes history from 1967 (before C was invented!) and has some good examples from Apache ("pools") and Postgres ("memory contexts").
I also just looked at these codebases:
https://github.com/mit-pdos/xv6-public (based on code from the 70's)
https://github.com/id-Software/DOOM (1997)
I looked at allocproc() in xv6, and gives you an object from a fixed global array. A lot of C code in the 80's and 90's was essentially "kernel code" in that it didn't have an OS underneath it. Embedded systems didn't run on full-fledges OSes.
DOOM tends to use a lot of what I would call "pools" -- arrays of objects of a fixed size, and that's basically what I remember from EA.
Though in g_game.c, there is definitely an arena of size 0x20000 called "demobuffer". It's used with a bump allocator.
---
So I'd say
- malloc / free of individual objects was NEVER what C code looked like (aside from toy code in college)
- arena allocators were used, but global vars and pools are also very common.
- arenas are more or less wash for memory safety. they help you in some ways, but hurt you in others.
The reason C programmers don't malloc/free all the time is for speed, not memory safety. Arenas are still unsafe.
When you free an arena, you have no guarantee there's nothing that points to it anymore.
Also, something that shouldn't be underestimated is that arena allocators break tools like ASAN, which use the malloc() free() interface. This was underscored to me by writing a garbage collector -- the custom allocator "broke" ASAN, and that was actually a problem:
https://www.oilshell.org/blog/2023/01/garbage-collector.html
If you want memory safety in your C code, you should be using ASAN (dynamically instrumented allocators) and good test coverage. Arenas don't help -- they can actually hurt. An arena is a trivial idea -- the problem is more if that usage pattern actually matches your application, and apps evolve over time.
- Run Linux Programs on DOS
-
The Magma operating system
Magma is proudly licensed under the MIT license, and uses code from Xv6 and Yagura.
-
User Space vs Kernel Space Development (For an experienced Dev)
My OS classes used xv6, a reimplementation of Unix Version 6 for a RISC-V architecture. Accompanying that was the OSTEP textbook.
-
MINIX is an awesome way to learn a wide range of CS concepts
Check out xv6 if you are only getting started with operating systems and want something simpler.
-
I am getting an undefined reference despite including the source file when compiling
Here is kernel.ld.
pgloader
-
Why Is Common Lisp Not the Most Popular Programming Language?
No, it's difficult to read, and understand. It's a parenthesis circus, example -
https://github.com/dimitri/pgloader/blob/master/src/sources/...
-
We need to talk about parentheses
Examples (for Common Lisp, so not citing Emacs): reddit v1, Google's ITA Software that powers airfare search engines (Kayak, Orbitz…), Postgres' pgloader (http://pgloader.io/), which was re-written from Python to Common Lisp, Opus Modus for music composition, the Maxima CAS, PTC 3D designer CAD software (used by big brands worldwide), Grammarly, Mirai, the 3D editor that designed Gollum's face, the ScoreCloud app that lets you whistle or play an instrument and get the music score,
but also the ACL2 theorem prover, used in the industry since the 90s, NASA's PVS provers and SPIKE scheduler used for Hubble and JWT, many companies in Quantum Computing, companies like SISCOG, who plans the transportation systems of european metropolis' underground since the 80s, Ravenpack who's into big-data analysis for financial services (they might be hiring), Keepit (https://www.keepit.com/), Pocket Change (Japan, https://www.pocket-change.jp/en/), the new Feetr in trading (https://feetr.io/, you can search HN), Airbus, Alstom, Planisware (https://planisware.com),
or also the open-source screenshotbot (https://screenshotbot.io), the Kandria game (https://kandria.com/),
and the companies in https://github.com/azzamsa/awesome-lisp-companies and on LispWorks and Allegro's Success Stories.
https://github.com/tamurashingo/reddit1.0/
http://opusmodus.com/
https://www.ptc.com/en/products/cad/3d-design
http://www.izware.com/mirai
https://apps.apple.com/us/app/scorecloud-express/id566535238
-
We migrated our PostgreSQL database with 11 seconds downtime
I worked on migrating our MySQL system to PostgreSQL using pgloader ( https://pgloader.io/ ).
There were some hiccups, things that needed clarification in documentation, and some additional processes that needed to be done outside of the system to get everything we need in place, it was a amazing help. Not sure the project would've been possible without it.
Data mapping from PostgreSQL to PostgreSQL as in the article isn't nearly as bad as going between systems. We took a full extended outage and didn't preload any data. There were many dry runs before hand and validation before hand, but the system wasn't so mission critical that we couldn't afford to shutoff the system for a couple of hours.
-
Time For Me To Fly… To Render
Initially, I started down the pgloader path, which seemed to be a common approach for the database conversion. However, using my M1-chip MacBook Pro led to some unexpected issues. Instead, I opted to use NMIG to convert MySQL to PostgreSQL. For more information, please check out the “Highlights From the Database Conversion” section below.
-
PostgresSQL DB restore sature whole SAN IO
I just checked pgloader (which is mostly for heterogeneous database migrations) and it doesn't seem to have any explicit throttling features, either. However, you can set concurrency, number of workers, and batch size.
-
What language should i learn?
But do what makes you happy. The best PostgreSQL data loader in the world is written in Common Lisp. Now, CL is a fast and ANSI standardized language with multiple implementations, and was used to create Reddit, but it's hard to call it fashionable.
-
Why Lisp?
- [ScoreCloud](https://scorecloud.com/) - A web and mobile application to automatically create music notation from music performance or recordings. Built with LispWorks.
## DB tools
- [Pgloader](https://github.com/dimitri/pgloader/) - Migrate to PostgreSQL in a single command!. [PostgreSQL License]
-
What ETL tool you use with Postgres ?
I would warmly recommend https://pgloader.io but, unfortunately, the Oracle support is still in need of a sponsor :-)
-
Installing pgLoader on RHEL9 - No matching package to install: 'sbcl' - is there a workaround?
I followed instruction here: https://github.com/dimitri/pgloader/blob/master/INSTALL.md, but can't get pass sbcl requirement.
-
Introducing pgsqlite, a pure python module for import sqlite into postgres
https://pgloader.io/ There's been a great tool available for this for many years.
What are some alternatives?
xv6-riscv - Xv6 for RISC-V
lisp-xl - Common Lisp Microsoft XLSX (Microsoft Excel) loader for arbitrarily-sized / big-size files
homebrew-i386-elf-toolchain - Homebrew formulas for buildling a valid GCC toolchain for the i386-elf target.
docker-postgres-upgrade - a PoC for using "pg_upgrade" inside Docker -- learn from it, adapt it for your needs; don't expect it to work as-is!
minixfromscratch - Development and compilation setup for the book versions of MINIX (2.0.0 and 3.1.0) on QEMU
HomeBrew - 🍺 The missing package manager for macOS (or Linux)
foam3 - FOAM: Feature-Oriented Active Modeller, Version 3 (unstable)
cl-wget - The Non-Interactive Network Downloader: cl-wget is a free software for retrieving files using HTTPS; cl-wget makes mirroring websites easy.
stumpwm - The Stump Window Manager
kandria - A post-apocalyptic actionRPG. Now on Steam!
lispe - An implementation of a full fledged Lisp interpreter with Data Structure, Pattern Programming and High level Functions with Lazy Evaluation à la Haskell.
TimescaleDB - An open-source time-series SQL database optimized for fast ingest and complex queries. Packaged as a PostgreSQL extension.