yugabyte-db VS Google Test

YugabyteDB is a transactional database that brings together four must-have needs of cloud native apps – namely SQL as a flexible query language, low-latency performance, continuous availability, and globally-distributed scalability. Other databases do not serve all 4 of these needs simultaneously.

It is possible to apply Batched Nested Loop but with additional code that checks the range of the outer bigint and compare it only if it matches the range of integer. This has been added in YugabyteDB 2.21 with #20715 YSQL: Allow BNL on joins over different integer types to help migrations from PostgreSQL with such datatype inconsistencies.

It can be done like https://github.com/yugabyte/yugabyte-db/ has.

Therefore, interleaving or co-partitioning is probably not necessary, and would reduce agility and scalability more than improving the performance. Unless you have a good reason for it that you can share on Issue #79. But, first, test and tune the queries to see if you need something else.

This is a workaround, not supported. I've opened the following issue to get it solve in the YugabyteDB deployment: https://github.com/yugabyte/yugabyte-db/issues/19389

Note that there may still be a need for bitmaps, especially with disjunctions (OR) as the following is about conjunction (AND), and it can still be implemented, differently than PostgreSQL. This is tracked by #4634.

FROM almalinux:8 as build RUN dnf -y update &&\ dnf groupinstall -y 'Development Tools' # get YugabyteDB sources ARG YB_TAG=2.18 RUN git clone --branch ${YB_TAG} https://github.com/yugabyte/yugabyte-db.git WORKDIR yugabyte-db # install dependencies and compilation tools RUN dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm RUN dnf -y install epel-release libatomic rsync python3-devel cmake3 java-1.8.0-openjdk maven npm golang gcc-toolset-12 gcc-toolset-12-libatomic-devel patchelf glibc-langpack-en ccache vim wget python3.11-devel python3.11-pip clang ncurses-devel readline-devel libsqlite3x-devel RUN mkdir /opt/yb-build RUN chown "$USER" /opt/yb-build # Install Python 3 RUN alternatives --remove-all python3 RUN alternatives --remove-all python RUN alternatives --install /usr/bin/python python /usr/bin/python3.11 3 RUN alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 3 # add #include "pg_yb_utils.h" to src/postgres/src/pl/plpython/plpy_procedure.c RUN sed -e '/#include "postgres.h"/a#include "pg_yb_utils.h"' -i src/postgres/src/pl/plpython/plpy_procedure.c # if using python > 3.9 remove #include and #include from src/postgres/src/pl/plpython/plpython.h RUN sed -e '/#include /d' -e '/#include /d' -i src/postgres/src/pl/plpython/plpython.h # add '--with-python', to python/yugabyte/build_postgres.py under the configure_postgres method RUN sed -e "/'\.\/configure',/a\ '--with-python'," -i python/yugabyte/build_postgres.py # Build and package the release RUN YB_CCACHE_DIR="$HOME/.cache/yb_ccache" ./yb_build.sh -j$(nproc) --clean-all --build-yugabyted-ui --no-linuxbrew --clang15 -f release RUN chmod +x bin/get_clients.sh bin/parse_contention.py bin/yb-check-consistency.py RUN YB_USE_LINUXBREW=0 ./yb_release --force WORKDIR / RUN mv /yugabyte-db/build/yugabyte*.tar.gz /yugabyte.tgz

You have seen me using the official YugabyteDB Docker image extensively. This image is suitable for various purposes, including labs, development, testing, and even production. In the past, we used to create it internally due to its seamless integration with our build process. However, some companies prefer to construct the image on their own, which is indeed a commendable practice. After all, it's not advisable to run random images with root privileges on your servers. As a result, we have made a significant alteration by introducing a refined Dockerfile to our Github repository.

Of course, I can guess from the function names, but YugabyteDB is Open Source and I can search for them. What happens here is that I didn't declare a Primary Key for my table and then an internal one (ybctid) is generated, because secondary indexes need a key to address the table row. This ID generation calls /dev/urandom. I made this simple example to show that low-level traces can give a clue about high level data model problems.

cmake_minimum_required(VERSION 3.5) project(knn_cpp CXX) include(FetchContent) FetchContent_Declare( googletest GIT_REPOSITORY https://github.com/google/googletest.git GIT_TAG release-1.11.0 ) FetchContent_MakeAvailable(googletest) FetchContent_Declare(matplotplusplus GIT_REPOSITORY https://github.com/alandefreitas/matplotplusplus GIT_TAG origin/master) FetchContent_GetProperties(matplotplusplus) if(NOT matplotplusplus_POPULATED) FetchContent_Populate(matplotplusplus) add_subdirectory(${matplotplusplus_SOURCE_DIR} ${matplotplusplus_BINARY_DIR} EXCLUDE_FROM_ALL) endif() function(knn_cpp_test TEST_NAME TEST_SOURCE) add_executable(${TEST_NAME} ${TEST_SOURCE}) target_link_libraries(${TEST_NAME} PUBLIC matplot) aux_source_directory(${CMAKE_CURRENT_SOURCE_DIR}/../lib LIB_SOURCES) target_link_libraries(${TEST_NAME} PRIVATE gtest gtest_main gmock gmock_main) target_include_directories(${TEST_NAME} PRIVATE ${CMAKE_CURRENT_SOURCE_DIR} ${CMAKE_CURRENT_SOURCE_DIR}/../) target_sources(${TEST_NAME} PRIVATE ${LIB_SOURCES} ) include(GoogleTest) gtest_discover_tests(${TEST_NAME}) endfunction() knn_cpp_test(LinearAlgebraTest la_test.cc) knn_cpp_test(KnnTest knn_test.cc) knn_cpp_test(UtilsTest utils_test.cc)

Okay, time to start unit tests!!! We will use Unity Test Framework to do unit testing. It is one of widely used testing frameworks alongside with Check, Google Test etc. Just downloading source code, and putting it to the project folder is enough to make it work (that is also why it is portable).

Updating GCC (it happened to GoogleTest).

Roy's project uses Google Test, a C++ testing framework. His testing setup is similar to mine as we both keep source files in one directory and tests in another. The key difference is that I can run the tests using the Visual Studios run button. It was fairly easy to write the new tests as there were existing ones that I could reference to check the syntax!

The Google Test Documentation provides a primer for first-time users. The primer introduces some basic concepts and terminology, some of which I've been able to learn for this lab exercise.

> git submodule add https://github.com/google/googletest.git > git submodule update --init --recursive

cmake_minimum_required(VERSION 3.14) project(my_project) # GoogleTest requires at least C++14 set(CMAKE_CXX_STANDARD 14) set(CMAKE_CXX_STANDARD_REQUIRED ON) include(FetchContent) FetchContent_Declare( googletest URL https://github.com/google/googletest/archive/03597a01ee50ed33e9dfd640b249b4be3799d395.zip ) # For Windows: Prevent overriding the parent project's compiler/linker settings set(gtest_force_shared_crt ON CACHE BOOL "" FORCE) FetchContent_MakeAvailable(googletest) enable_testing() add_executable( hello_test hello_test.cpp ) target_link_libraries( hello_test GTest::gtest_main ) include(GoogleTest) gtest_discover_tests(hello_test)

FetchContent\_Declare(googletestGIT\_REPOSITORY [[email protected]](mailto:[email protected]):googletest.git [https://github.com/google/googletest.git](https://github.com/google/googletest.git)GIT\_TAG release-1.12.1)FetchContent\_MakeAvailable(googletest)

Not sure about CppUnit but I can speak to my previous experience using the googletest framework which compiles your tests to an executable, and since it's a very simple framework we were able to cross-compile and run directly on our device. We just had to hook up a device to the server that was running the CI so it could flash it when needed. That basically meant that our process was: