XTOS / OPEN SOURCE REAL-TIME SYSTEMOPEN SOURCE · 1.0.2 · PRERELEASE · IMAGES & SOURCE
DocsTroubleshooting

Troubleshooting

Use symptoms, check results, and repair paths for build, guest, and BCI failures.

On this page

Error records

First distinguish a builder, host verifier, and guest test failure. Host uname does not prove that the QEMU guest uses PREEMPT_RT. .build is used only when BUILD_ROOT is unset. Do not delete it first.

Host: retain the verifier status instead of replacing it with a later command.
git rev-parse HEAD
git status --short
uname -a
gcc --version
make --version
BUILD_ROOT="${BUILD_ROOT:-$PWD/.build}"
printf 'build_root=%s\n' "$BUILD_ROOT"
if QEMU_TIMEOUT=300 ./scripts/verify_system.sh; then
  verify_status=0
else
  verify_status=$?
fi
printf 'verify_exit=%s\n' "$verify_status"
SymptomCheck resultRepair
verify_system.sh returns 1Read system_qemu=fail exit=… and system-qemu.log; 1 is the verifier status, not always the raw QEMU status.Repair the boot, marker, or panic shown in the log.
timeout returns 124The budget expired; inspect progress and final markers.Locate the stall before increasing the budget.
Guest returns 1Read missing markers and Failed; the script ignores the QEMU pipeline status.Repair the first missing or failed check.
Basic cross-architecture status 0The basic script allows up to two FAIL results.Run verify_guest_* and require Failed: 0.

Dependencies and resources

For command not found, missing headers, or link failures, check PATH and development packages first. build_system.sh checks only some tools. A killed cc1 usually indicates memory pressure; “No space left on device” indicates disk exhaustion, and a source syntax error points to compiler output.

Host: check tools required by the target architecture.
for tool in gcc make flex bison bc cpio gzip rsync python3 depmod; do
  command -v "$tool" || printf 'missing=%s\n' "$tool"
done
df -h .
free -h
command -v qemu-system-x86_64
command -v aarch64-linux-gnu-gcc
command -v riscv64-linux-gnu-gcc
test -r /usr/share/qemu/opensbi-riscv64-generic-fw_dynamic.bin
Missing itemDebian/Ubuntu packageResult and repair
Compiler and generatorsbuild-essential flex bison bcRerun the failed stage after installation; use JOBS=2 when memory is low.
OpenSSL, ELF, libc development fileslibssl-dev libelf-dev libc6-devKeep the compiler diagnostic; a header error is not a QEMU error.
Packing and module toolscpio gzip kmod rsync python3 filex86 uses cpio --reproducible; check that the option exists first.
x86_64 QEMUqemu-system-x86qemu-system-x86_64 is required.
ARM64crossbuild-essential-arm64 libc6-dev-arm64-cross qemu-system-armRequires aarch64-linux-gnu-gcc and qemu-system-aarch64.
RISC-V64gcc-riscv64-linux-gnu binutils-riscv64-linux-gnu libc6-dev-riscv64-cross qemu-system-misc opensbiCheck qemu-system-riscv64 and the fixed OpenSBI path.
Host: retry x86 with lower parallelism after fixing dependencies; check free disk and memory first.
cpio --help | grep -- --reproducible
JOBS=2 ./scripts/build_system.sh

Images and timeouts

For missing images, complete the matching build or point the verifier at the real BUILD_ROOT. On timeout, check whether the log reached the kernel, init, tests, or shutdown. An ordinary cross-architecture image enters a BusyBox shell; only the guestcheck image validates and powers off.

Host: inspect x86 artifacts and logs; grep is diagnostic only.
BUILD_ROOT="${BUILD_ROOT:-$PWD/.build}"
for image in xtos-bzImage xtos-rootfs.cpio.gz kernel.config; do
  test -s "$BUILD_ROOT/images/$image" || printf 'missing_or_empty=%s\n' "$image"
done
grep -nE 'Linux version|Run /init|realtime=|xtos_modules=|guest_|XTOS_|Kernel panic|Power down|power down|error:' "$BUILD_ROOT/system-qemu.log"
SymptomCheckRepair
No Linux versionQEMU, image, architecture; also OpenSBI for RISC-V64.Fix the file or command and rerun the same architecture.
Kernel but no STARTCheck the guestcheck initramfs; ARM64 uses ttyAMA0 and RISC-V64 ttyS0.Run verify_guest_*; ordinary shell boot is not guest validation.
START but no test PASSCheck test_*, module errors, and QEMU progress.Restore tests or fix assertions; increase the budget only for a progressing slow run.
Only DONECheck realtime, configuration, modules, each test, and FAIL.DONE is not a pass; retain the log and fix the specific item.
Host: rerun strict guest checks; QEMU_MEM defaults to 2G and QEMU_SMP to 2.
QEMU_TIMEOUT=300 ./scripts/verify_guest_arm64.sh
QEMU_TIMEOUT=360 ./scripts/verify_guest_riscv64.sh

Real-time configuration

Run xtos-rt-status in the guest: 1/exit 0 confirms its RT checks, 0/exit 1 means an existing node did not confirm RT, and unknown/exit 2 means unconfirmed. If /sys/kernel/realtime is absent, it requires both CONFIG_PREEMPT_RT=y in the running kernel’s /proc/config.gz and PREEMPT_RT in uname -v. Build configuration cannot replace running configuration.

Guest: inspect the running kernel and mounts.
uname -r
cat /proc/mounts
/usr/bin/xtos-rt-status
zcat /proc/config.gz | grep -E '^CONFIG_(PREEMPT_RT|XTOS_SCHED|XTOS_BCI|XTOS_BCI_OPENBCI|IKCONFIG|IKCONFIG_PROC)='
Host ARM64: x86 uses kernel-build and RISC-V64 uses kernel-build-riscv64.
BUILD_ROOT="${BUILD_ROOT:-$PWD/.build}"
grep -E '^CONFIG_(PREEMPT_RT|XTOS_SCHED|XTOS_BCI|XTOS_BCI_OPENBCI|IKCONFIG|IKCONFIG_PROC)=' "$BUILD_ROOT/kernel-build-arm64/.config"
grep -E '^CONFIG_(PREEMPT_RT|XTOS_SCHED|XTOS_BCI|XTOS_BCI_OPENBCI)=' "$BUILD_ROOT/kernel-build-arm64/include/config/auto.conf"

Expected values are PREEMPT_RT=y, XTOS_SCHED=y, XTOS_BCI=m, XTOS_BCI_OPENBCI=m, IKCONFIG=y, and IKCONFIG_PROC=y. Cross builders initialize configuration only when .config is absent, so an old BUILD_ROOT can retain wrong settings.

Host: rebuild ARM64 in a new directory without deleting old artifacts.
repair_build=$(mktemp -d "$HOME/xtos-repair-arm64.XXXXXX")
BUILD_ROOT="$repair_build" JOBS=2 ./scripts/build_system_arm64.sh
BUILD_ROOT="$repair_build" QEMU_TIMEOUT=300 ./scripts/verify_guest_arm64.sh

Module versions

The modprobe names are xtos_bci_core, xtos_bci_buffer, and openbci_cyton, not CONFIG_XTOS_BCI. A .ko in a build directory does not prove a matching module in guest /lib/modules. Load checks run in the target guest.

Guest: retain modprobe stderr and dmesg.
uname -r
find /lib/modules -maxdepth 2 -type f -name modules.dep -print
find /lib/modules -type f -name '*xtos*' -print
find /lib/modules -type f -name '*openbci*' -print
modprobe xtos_bci_core
modprobe xtos_bci_buffer
modprobe openbci_cyton
dmesg | tail -80
ls -d /sys/module/xtos_bci_core /sys/module/xtos_bci_buffer /sys/module/openbci_cyton
ErrorCheck and repair
module not foundCompare uname -r with /lib/modules; confirm modules_install and depmod, then rebuild matching kernel/rootfs.
invalid module formatUse host modinfo for vermagic and compare with the guest release; do not force-load.
Unknown symbolKeep symbol names from dmesg; check dependencies, indexes, and mixed .ko files.
Operation not permittedCheck id in the guest; host root does not grant guest-process privileges.
Host ARM64: compare installed module metadata with kernel.release from the same build.
BUILD_ROOT="${BUILD_ROOT:-$PWD/.build}"
find "$BUILD_ROOT/rootfs-arm64/lib/modules" -type f -name 'xtos_bci_core.ko*' -exec modinfo -F vermagic {} \;
cat "$BUILD_ROOT/kernel-build-arm64/include/config/kernel.release"

Use the system builder to install modules, generate depmod indexes, and pack initramfs. Do not copy one .ko into another release directory; build matching kernel and rootfs in a new BUILD_ROOT.

Library architecture

file usually reports only current ar archive for .a files. For file in wrong format, inspect archive members and use the target objdump on the three libraries under rootfs/usr/lib.

Host: expect architecture: aarch64 for ARM64 and architecture: riscv:rv64 for RISC-V64.
BUILD_ROOT="${BUILD_ROOT:-$PWD/.build}"
aarch64-linux-gnu-objdump -f "$BUILD_ROOT/rootfs-arm64/usr/lib/libxtos-rt.a"
aarch64-linux-gnu-objdump -f "$BUILD_ROOT/rootfs-arm64/usr/lib/libxtos-signal.a"
aarch64-linux-gnu-objdump -f "$BUILD_ROOT/rootfs-arm64/usr/lib/libxtos-bci.a"
riscv64-linux-gnu-objdump -f "$BUILD_ROOT/rootfs-riscv64/usr/lib/libxtos-rt.a"
TargetExpected objdumpCompiler
ARM64architecture: aarch64aarch64-linux-gnu-gcc
RISC-V64architecture: riscv:rv64riscv64-linux-gnu-gcc
Host: export an independent ARM64 copy from the current commit; use matching scripts for RISC-V64.
repair_source=$(mktemp -d "$HOME/xtos-archive-repair.XXXXXX")
git archive HEAD | tar -x -C "$repair_source"
cd "$repair_source"
JOBS=2 ./scripts/build_system_arm64.sh
QEMU_TIMEOUT=300 ./scripts/verify_guest_arm64.sh

ARM64/RISC-V64 builders rebuild SDK libraries in source directories, so a new BUILD_ROOT does not isolate those artifacts.

Use separate source copies and build serially when retaining both architectures.

If libraries are correct but tests are missing, inspect Stage 4 warnings; cross builders may continue packing, while strict guest reports guest_test_*=missing.

BCI nodes

Check module loading, driver binding, device registration, and session control separately. examples/bci_test.c defaults to /dev/xtos0 and accepts a path as its first argument. The default string does not prove a device exists; an ordinary serial port or empty manual node is not an XTOS BCI device.

Target system: inspect nodes, permissions, modules, and binding logs; do not run acquisition examples without hardware.
id
ls -l /dev/xtos* /dev/ttyUSB* /dev/ttyACM* 2>/dev/null
find /sys/class -maxdepth 2 -iname '*bci*' -print
ls -d /sys/module/xtos_bci_core /sys/module/xtos_bci_buffer /sys/module/openbci_cyton
dmesg | tail -100
ErrorCauseRepair
ENOENTThe path does not exist; ordinary QEMU has no physical BCI.Check the registered node and binding; record connectivity as unverified without hardware.
EACCES / EPERMsession_create uses O_RDWR and needs read and write access.Grant access by owner, group, and process identity; chmod 777 is not a general fix.
ENOTTYThe file or driver does not support the BCI ioctl, or the request encoding differs; a serial port does not implement XTOS ioctls.Check device type and kernel/SDK UAPI; sudo cannot add an ioctl.
Modules but no nodeModule presence does not prove serdev binding or device registration.Check board serial, device tree, and driver logs; physical-device bring-up is required.
UAPI: compare the magic, request numbers, and type sizes.
#define XTOS_BCI_GET_INFO _IOR(XTOS_BCI_IOC_MAGIC, 1, struct xtos_bci_info)
#define XTOS_BCI_START_STREAM _IO(XTOS_BCI_IOC_MAGIC, 2)
#define XTOS_BCI_CHECK_IMP _IOR(XTOS_BCI_IOC_MAGIC, 8, uint32_t[32])

v1.0.2 read_samples numerically converts driver int32 nanovolts to float microvolts, accumulates reads until the requested bytes or EOF, and retries EINTR. Allocation-size overflow gives EOVERFLOW, malloc failure normally ENOMEM, and a trailing partial integer at EOF gives EIO. Other read errors, including EAGAIN, return -1 without delivering data already consumed by the call.

read_samples uses malloc/free on each call and is not allocation-free real-time acquisition; it returns EBUSY while a frame snapshot is held or awaiting cleanup. Calibration and impedance return EOPNOTSUPP. Cyton supports only 250 Hz, with register readback for channel gain; unsupported higher rates also return EOPNOTSUPP.

Legacy get_mmap_buffer returns NULL and size=0, with EOPNOTSUPP for a valid session. Use versioned snapshot/config/frame interfaces, consume by token/cursor, and retry failed releases. Externally serialize a session. Frame host_rx_ns is not device acquisition time, and added metadata is not physical-board acceptance.

Tests and reports

All three SDKs have public test targets. BCI runs test_samples, test_snapshot, test_snapshot_uapi, test_config, and test_frame for software conversion, snapshot lifecycle, UAPI layout, configuration validation, and frame views. FIFO or syscall fixtures do not validate physical devices or exhaust every error path.

Host: build SDKs for the native architecture; includes the BCI software-conversion test and ABI metadata check.
make -C sdk/libxtos-rt all test
make -C sdk/libxtos-signal all test
make -C sdk/libxtos-bci all test
./scripts/check_abi.sh
Inspect staged SDKs only after the x86 build_system.sh completes.
XTOS_ABI_SDK_ROOT="${BUILD_ROOT:-$PWD/.build}/userspace-source/sdk" ./scripts/check_abi.sh

check_abi.sh defaults to shared libraries under source sdk/; XTOS_ABI_SDK_ROOT selects another SDK root. x86 build_system.sh stages libraries under BUILD_ROOT/userspace-source/sdk/, allowing checks without rebuilding in the source tree.

no built shared object found means the selected SDK root has no shared objects; complete its build or fix the path without mixing target architectures.

The script checks SONAME and filename major/minor, not full layout or symbol compatibility.

  • Report the host OS, source commit, worktree changes, and target architecture.
  • Report compiler and QEMU versions, exact commands, environment variables, and script status.
  • Attach full error output and serial logs, and identify the first missing or failed marker.
  • State the new build directory and whether physical hardware exists and is bound.

Source: public build and verifier scripts, sdk/libxtos-bci/Makefile and src/session.c, examples/bci_test.c, kernel and SDK BCI UAPI, scripts/check_abi.sh, and public Makefiles.