QUARK(7) Miscellaneous Information Manual QUARK(7)

quark — unified system process telemetry library

TABLE OF CONTENTS

quark is a library that provides a way to retrieve and listen to process events in linux systems. Its main purpose is to abstract different backends and to provide a common API for listening to system-wide events like fork(2), exec(3), exit(3) and others.

quark not only provides an API for listening to events, but also handles ordering, buffering and aggregation of said events. In its most basic form, a short lived process consisting of fork(2) + exec(3) + exit(3) will be aggregated into one quark_event. An internal process cache is also kept that can be looked up via quark_process_lookup(3).

Clone the repository, compile and run quark's test utility quark-mon(8):

$ git clone --recursive https://github.com/elastic/quark
$ cd quark
$ make
$ sudo ./quark-mon

On another shell, create any process like:
$ ls -1 /tmp | wc -l

See BUILDING for a list of dependencies if you're having trouble building. Also see INCLUDED BINARIES and quark-mon(8).

quark tries to guarantee event ordering as much as possible. Ordering must be done in userland for some backends, notably anything that uses perf-rings. quark uses two for ordering and aggregation.

The first tree is basically a priority queue, ordered by the time of the event. The second tree is ordered by time of the event + pid and it's used for event aggregation.

quark buffers and aggregates related events that happened close enough. The common case is generating a single event for the triple: fork(2), exec(3), exit(3). There are rules on what can be aggregated, and only events of the same pid are aggregated. For example: quark won't aggregate two exec(3) events, otherwise we would lose the effects of the first one. These rules will be exposed and configurable in the future.
For aggregation and ordering to work, quark needs to be able to buffer events, this means holding them before presenting them to the user. quark employs an ageing timeout that is a stepped function of the number of currently buffered events, the more events you have, the shorter the timeout will be, so memory can be bound. A quark_event is only given to the user when it has a certain age. From quark.c: 
/*
 * Target age is the duration in ns of how long should we hold the event in the
 * tree before processing it. It's a function of the number of items in the tree
 * and its maximum capacity:
 * from [0; 10%]    -> 1000ms
 * from [90%; 100%] -> 0ms
 * from (10%; 90%)  -> linear from 1000ms -> 100ms
 */
The library tries to give as much context for an event as possible. Depending on the backend, the events we read from the kernel can be limited in context. quark maintains an internal process table with what has been learned about the process so far, this context is then included in each event given to the user. The process table can also be queried, see below.
An internal cache of processes is kept that can be looked up via quark_process_lookup(3). This cache keeps soon-to-be-purged elements for a little while so that you can still lookup a process that just exited. The table is initialized by scraping /proc.
quark tries to be as transparent as possible about what it knows, there are counters for lost events, and each piece of information of a quark_event is guarded by a flag, meaning the user might get incomplete events in the case of lost events, it's the user responsability to decide what to do with it.

Depending on load, the user might see an event as the aggregation of multiple events, or as independent events. The content remains the same.

quark is written in C, but Go bindings are also provided. Ideally we will be able to provide bindings for other languages in the future.
Currently, EBPF and a kprobe-based backend are provided, but we would like to add AUDIT support as well. The backend in use is transparent to the user and unless specified, quark will try to use the EBPF, falling back to KPROBE if it failed.

quark can be built natively or via a container, native is preferred and depends on:

Make sure to clone the repository recursively: .

make builds the repository, including quark-mon, libquark_big.a and a libquark.a.

libquark_big.a includes all needed dependencies in one big archive. This includes a libbpf.a, libelf_pic.a (from the elftoolchain project, BSD license), and a libz.a (see zlib/LICENSE). See LINKING to learn how to link either.

While quark doesn't build elastic/ebpf, it does use the EBPF programs from that repository, only the files needed are included in quark, as elastic/ebpf is quite big.

Other useful build targets include:

Clean object files from quark.
Builds quark inside a docker container, so you don't have to worry about having build dependencies.
Builds quark for arm64 inside a docker container.
Builds quark inside a centos7 docker container, useful for linking against ancient glibc-2.17.
Builds and runs quark-test(8).
Runs quark-test(8) over all kernels in kernel_images/.
Shortcut for test + test-kernels.
Regenerates btfhub.c. Usage: 
$ make btfhub BTFHUB_ARCHIVE_PATH=/my/path/to/btfhub-archive
Clean all object files, including the ones from , and .
Lints and generates all the documentation from manpages in docs/.
Builds an SVG out of the DOT files produced by quark-mon(8).
Generates README.md out of quark.7.
Copies the files from EEBPF_PATH used by quark. Usage: 
$ make eebpf-sync EEBPF_PATH=/my/path/to/elastic/ebpf
Builds an initramfs file containing all quark binaries so that it can be run as the init process on boot, useful for testing any kernel under qemu. See TESTING.

All the targets above can generate debug output by specifying , as in:

$ make V=1

$ cc -o myprogram myprogram.c libquark_big.a
OR
$ cc -o myprogram myprogram.c libquark.a libbpf/src/libbpf.a elftoolchain/libelf/libelf_pic.a zlib/libz.a

quark-test(8) is the main test utility ran by the CI, can be invoked via make test. All tests are self-contained in this binary.

Some included kernels can be tested in qemu via make test-kernel. Any quark utility can be run on a custom kernel via the krun.sh script, as in:

$ make initramfs.gz
$ ./krun.sh initramfs.gz kernel-images/amd64/linux-4.18.0-553.el8_10.x86_64 quark-test -vvv

Note that you can pass arguments to the utility and you have to make initramfs.gz first.

quark-mon(8) is a program that dumps quark_events to stdout and can be used for demo and debugging. It has a neat feature: can be run without priviledges, while useless in this small program, it aims to demonstrate how a user could implement the same.

quark-btf(8) is a program for dumping BTF information used by quark.

quark-test(8) is a program for running tests during development.

The ball starts with quark_queue_open(3).

quark_queue_open(3) initializes a quark_queue which holds the majority of runtime state used by library, this includes perf-rings, file descriptors, EBPF programs buffering data-structures and the like. It must be paired with a quark_queue_close(3) on exit.

quark_queue_get_events(3) is the main driver of the library, it does the buffering, per-ring scanning, aggregation and event cache garbage collecting. In case there are no events it returns zero and the user is expected to call quark_queue_block(3) or equivalent.

#include <err.h>
#include <quark.h>
#include <stdio.h>

int
main(void)
{
	struct quark_queue	qq;
	struct quark_event	qevs[32], *qev;
	int			n, i;

	if (quark_queue_open(&qq, NULL) == -1)
		err(1, "quark_queue_open");

	for (; ;) {
		n = quark_queue_get_events(&qq, qevs, 32);
		if (n == -1) {
			warn("quark_queue_get_events");
			break;
		}
		/* Scan each event */
		for (i = 0, qev = qevs; i < n; i++, qev++)
			quark_event_dump(qev, stdout);
		if (n == 0)
			quark_queue_block(&qq);
	}

	quark_queue_close(&qq);

	return (1);
}

quark_queue_open(3)
open a queue to receive events, initial library call.
quark_queue_default_attr(3)
get default attributes of quark_queue_open(3).
quark_queue_get_events(3)
get events, main library call.
quark_process_lookup(3)
lookup a process in quark's internal cache
quark_event_dump(3)
dump event, mainly a debugging utility.
quark_queue_get_epollfd(3)
get a descriptor suitable for blocking.
quark_queue_block(3)
block for an unspecified amount of time.
quark_queue_get_stats(3)
basic queue statistics.
quark_queue_close(3)
close a queue.

quark_queue_get_events(3) is the meat of the library and contains further useful documentation.

quark-mon(8) is the easiest way to get started with quark.

quark_queue_open(3) describes initialization options that can be useful.

quark_event_dump(3), quark_process_lookup(3), quark_queue_block(3), quark_queue_close(3), quark_queue_get_epollfd(3), quark_queue_get_events(3), quark_queue_get_stats(3), quark_queue_open(3), quark-btf(8), quark-mon(8), quark-test(8)

quark is released under the Apache-2.0 license and contains code under BSD-2, BSD-3, ISC, and zlib Licenses.

quark started in April 2024.

November 12, 2024 Linux