moja VS Knapsack

I want sum types.

I want a statically-typed way of constructing composable queries that follow SQL rather than reinvent a different thing. It doesn't have to be the same syntax but it has to be the same structuring.

I started writing one[0] and stopped before doing all the boilerplate code generation, having moved on from the JVM ecosystem for the time being. One thing it does is treat most things like sets so we don't end up with N+1 queries. Another trick it uses is collapsing constant expressions via an expression evaluation library[1].

[0] https://github.com/karmakaze/safeql

[1] https://github.com/karmakaze/moja

Over the years, I've written many apps and utilities for myself or others (that didn't end up get used). These are the interesting ones I remember. Many not quite complete/usable. Other than hackerer.news none of them are 'up' and running. Some have and others haven't been published as opensource.

- https://hackerer.news HN viewer (source[0]): I use daily so I can see today's top stories in reverse chronological order with mainstream topics sorted to the bottom.

- qwickly[1] keyboard layout: I use all the time as an easier to learn and more comfortable to type than Colemak/Tarmak

- safeql[2]: Java type-safe SQL expression composer that reduces constant expressions and eliminates N+1 queries loading associations by always operating on set relation or array of models.

- moja[3]: Composable computation pipelines for Java: Async, Lazy, Option, Try, Result, Multi (List), Stated, Reader, Logger, Writer.

- gitgrep.com[4] Opensource SaaS version of etsy/houndd (now called hound-search).

- statuspages.me: Status page aggregator with dynamic javascript for scraping each source using selector expressions.

- movies to watch aggregator: with links to sources to watch. It was hard then to get 3rd party deep links into streaming sites so included some torrent links. Got a DMCA phone call, so took it down. Combined thumbnails, summaries, actors(?), imdb ratings, links.

- java2cpp: Translate a moderately sized java app with test suite to c++, not 100% required final manual fixups.

- swift2java (or maybe it was java2swift, it's fuzzy now): translate Swift to Java obviously, using ANTLR4. Not 100% required final manual fixups.

- gui2log: to make an ASCII rendition of on-screen GUI widgets into an application log file when form submitted, so users couldn't complain that they saw X, but got Y.

- some basic stats/ML algorithms: k-nearest neighbour, RNN back-propagation, etc?

- Java in-memory DB: Small SQL-like memory tables with indexing/searching.

- wwwsqldesigner: This exists as opensource and I extended it to infer foreign key relationships based on naming conventions used in a MySQL schema. It was great for zooming around a large ERD.

- tracelog: combination of microservices parent/child span logging and generated high level events shown as a sequence diagram. Integrated with Loggly for full/verbose logs of selected high-level events.

- pcl2bmp downscaler: Reduce high resolution HP LaserJet (PCL5) printed to file to lower resolution bitmap pages for screen display (before retina DPI was common). It aimed to shrink same-color areas and preserve black/white transitions while reducing.

[0] https://gitlab.com/karmakaze/hackerer-news

[1] https://github.com/qwickly-org/Qwickly

[2] https://github.com/karmakaze/safeql

[3] https://github.com/karmakaze/moja

[4] https://github.com/gitgrep-com/gitgrep

Sure, SafeQL[0] SQL library. It's been sitting in a close but not enough to promote state for a while. The main thing I wanted was to have generation of the bindings to existing DB schema. I also want to combine using Moja[1] datatypes for uniform handling of single/multi, normal/async, and errors.

[0] https://github.com/karmakaze/safeql

[1] https://github.com/karmakaze/moja

I made a similar thing for Java moja Options as needed other projects. It sort-of works but it's not great.

I've created a knapsack ruby gem for CI parallelisation that has over 122 million downloads. Primarily due to the fact, Gitlab is using it.

I spin off https://knapsackpro.com from the knapsack gem and we are helping our customers run fast CI builds.

name: Main on: [push] jobs: test: runs-on: ubuntu-latest # If you need DB like PostgreSQL, Redis then define service below. # https://github.com/actions/example-services/tree/master/.github/workflows services: postgres: image: postgres:10.8 env: POSTGRES_USER: postgres POSTGRES_PASSWORD: "" POSTGRES_DB: postgres ports: - 5432:5432 # needed because the postgres container does not provide a healthcheck # tmpfs makes DB faster by using RAM options: >- --mount type=tmpfs,destination=/var/lib/postgresql/data --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5 redis: image: redis ports: - 6379:6379 options: --entrypoint redis-server # https://help.github.com/en/articles/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix strategy: fail-fast: false matrix: # [n] - where the n is a number of parallel jobs you want to run your tests on. # Use a higher number if you have slow tests to split them between more parallel jobs. # Remember to update the value of the `ci_node_index` below to (0..n-1). ci_node_total: [8] # Indexes for parallel jobs (starting from zero). # E.g. use [0, 1] for 2 parallel jobs, [0, 1, 2] for 3 parallel jobs, etc. ci_node_index: [0, 1, 2, 3, 4, 5, 6, 7] env: RAILS_ENV: test GEMFILE_RUBY_VERSION: 2.7.2 PGHOST: localhost PGUSER: postgres # Rails verifies the time zone in DB is the same as the time zone of the Rails app TZ: "Europe/Warsaw" steps: - uses: actions/checkout@v2 - name: Set up Ruby uses: ruby/setup-ruby@v1 with: # Not needed with a .ruby-version file ruby-version: 2.7 # runs 'bundle install' and caches installed gems automatically bundler-cache: true - name: Create DB run: | bin/rails db:prepare - name: Run tests env: KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC: ${{ secrets.KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC }} KNAPSACK_PRO_CI_NODE_TOTAL: ${{ matrix.ci_node_total }} KNAPSACK_PRO_CI_NODE_INDEX: ${{ matrix.ci_node_index }} KNAPSACK_PRO_LOG_LEVEL: info # if you use Knapsack Pro Queue Mode you must set below env variable # to be able to retry CI build and run previously recorded tests # https://github.com/KnapsackPro/knapsack_pro-ruby#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node KNAPSACK_PRO_FIXED_QUEUE_SPLIT: true # RSpec split test files by test examples feature - it's optional # https://knapsackpro.com/faq/question/how-to-split-slow-rspec-test-files-by-test-examples-by-individual-it KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES: true run: | bundle exec rake knapsack_pro:queue:rspec

# Ruby CircleCI 2.0 configuration file # # Check https://circleci.com/docs/2.0/language-ruby/ for more details # version: 2 jobs: build: parallelism: 10 # https://circleci.com/docs/2.0/configuration-reference/#resource_class resource_class: small docker: # specify the version you desire here - image: circleci/ruby:2.7.1-node-browsers environment: PGHOST: 127.0.0.1 PGUSER: my_db_user RAILS_ENV: test # Split slow RSpec test files by test examples # https://knapsackpro.com/faq/question/how-to-split-slow-rspec-test-files-by-test-examples-by-individual-it KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES: true # Specify service dependencies here if necessary # CircleCI maintains a library of pre-built images # documented at https://circleci.com/docs/2.0/circleci-images/ - image: circleci/postgres:10.6-alpine-ram environment: POSTGRES_DB: my_db_name POSTGRES_PASSWORD: "" POSTGRES_USER: my_db_user # Rails verifies Time Zone in DB is the same as time zone of the Rails app TZ: "Europe/Warsaw" - image: redis:6.0.7 working_directory: ~/repo environment: TZ: "Europe/Warsaw" steps: - checkout # Download and cache dependencies - restore_cache: keys: - v2-dependencies-bundler-{{ checksum "Gemfile.lock" }}-{{ checksum ".ruby-version" }} # fallback to using the latest cache if no exact match is found - v2-dependencies-bundler- - run: name: install ruby dependencies command: | bundle install --jobs=4 --retry=3 --path vendor/bundle - save_cache: paths: - ./vendor/bundle key: v2-dependencies-bundler-{{ checksum "Gemfile.lock" }}-{{ checksum ".ruby-version" }} # Database setup - run: bin/rails db:prepare - run: name: run tests command: | export CIRCLE_TEST_REPORTS=/tmp/test-results mkdir $CIRCLE_TEST_REPORTS bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec.xml]" # collect reports - store_test_results: path: /tmp/test-results - store_artifacts: path: /tmp/test-results destination: test-results

One, the interesting out-of-the-box solution is Knapsack Pro. It helps run your tests in a parallel efficient way.