mORMot Open Source

zen010101

Member

Registered: 2024-06-15

Posts: 171

EInvalidOp crash on aarch64-linux when C library calls mORMot's export

Environment
- Platform: aarch64-linux (ARM64, Rockchip RK3588)
- FPC: 3.2.2
- mORMot2: latest
- Application: FPC program linked with GStreamer + FFmpeg (libavcodec)

Symptom
Program crashes with EInvalidOp: Invalid floating point operation when FFmpeg's libavcodec calls pow() during audio encoder initialization.

GDB Backtrace

Thread 28 "queue210:src" hit Breakpoint 1, SYSTEM_$_FLOAT_RAISE$TFPUEXCEPTIONMASK ()
(gdb) bt
#0  SYSTEM_$_FLOAT_RAISE$TFPUEXCEPTIONMASK ()
#1  SYSTEM_$_FLOAT_RAISE$TFPUEXCEPTION ()
#2  SYSTEM_$_RAISEPENDINGEXCEPTIONS ()
#3  SYSTEM_$_FPC_THROWFPUEXCEPTION ()
#4  fpc_ln_real ()
#5  MATH_$_POWER$DOUBLE$DOUBLE$DOUBLE ()
#6  p0w (b=-18014398509481984, e=0) at mormot.lib.static.pas:1014
#7  () at /lib/aarch64-linux-gnu/libavcodec.so.58
...
#10 avcodec_open2 () at /lib/aarch64-linux-gnu/libavcodec.so.58

Root Cause Analysis

The issue involves three interacting factors:

1. Symbol Export in mormot.lib.static.pas

mORMot exports C-compatible math functions with standard libc symbol names:

function p0w(b, e: double): double; cdecl; export alias: 'pow';
begin
  result := Power(b, e);  // calls FPC's Math.Power
end;

When the FPC program is linked, these symbols override the libc versions. Any C library (like libavcodec) calling pow() will resolve to mORMot's p0w instead of glibc's implementation.

2. FPC's Soft-Float Exception Handling on aarch64

On aarch64, FPC uses software-based floating-point exception detection. The FPU exception state is stored in a ThreadVar:

// From FPC RTL system unit
threadvar
  softfloat_exception_flags: TFPUExceptionMask;
  softfloat_exception_mask: TFPUExceptionMask;

After each floating-point operation, FPC checks softfloat_exception_flags and raises Pascal exceptions if needed.

3. ThreadVar Initialization in Foreign Threads

The critical issue: ThreadVars are only initialized for threads created by FPC's BeginThread.

When a C library (GStreamer/libavcodec) creates its own threads using pthread_create, FPC's ThreadVar storage is NOT initialized. The softfloat_exception_mask contains garbage/zero instead of the default mask that suppresses common exceptions.

Call Flow

1. GStreamer creates worker thread via pthread_create
   -> FPC ThreadVars NOT initialized (softfloat_exception_mask = 0 or garbage)

2. libavcodec calls pow() for audio encoding setup
   -> Resolves to mORMot's p0w() due to symbol export

3. p0w() calls FPC's Power() -> calls fpc_ln_real()
   -> FPC soft-float code checks softfloat_exception_flags
   -> Uninitialized mask causes false positive exception detection

4. FPC raises EInvalidOp in a non-FPC thread
   -> Crash

Current Workaround

Define NOLIBCSTATIC to disable all C library function exports:

-dNOLIBCSTATIC

This works but disables ALL static C functions, which may be too aggressive.

Suggested Improvement

Rather than disabling all exports, consider a more targeted approach:

Option A: Separate define for math function exports

{$ifndef NOLIBCMATH}
function p0w(b, e: double): double; cdecl; export alias: 'pow';
function fl00r(x: double): double; cdecl; export alias: 'floor';
function ce1l(x: double): double; cdecl; export alias: 'ceil';
// ...
{$endif}

Option B: Make these functions thread-safe for foreign threads

Check if running in a properly initialized FPC thread before using FPC math:

function p0w(b, e: double): double; cdecl; export alias: 'pow';
begin
  // If ThreadVar not initialized, fall back to libc or use safe implementation
  if not IsFPCThreadInitialized then
    result := libc_pow(b, e)  // direct syscall or dlsym
  else
    result := Power(b, e);
end;

Option C: Document the limitation

At minimum, document that programs using C libraries that create threads (GStreamer, FFmpeg, SDL, etc.) should define NOLIBCSTATIC on aarch64.

Chaa

Member

Registered: 2011-03-26

Posts: 261

Re: EInvalidOp crash on aarch64-linux when C library calls mORMot's export

Program crashes with EInvalidOp: Invalid floating point operation when FFmpeg's libavcodec calls pow() during audio encoder initialization.

This is expected behavior. C and Pascal uses different floating point settings.

See SetFpuFlags from mormot.lib.static.pas and TSynFPUException in mormot.core.perf.pas.

You can permanently set floating point settings for specific thread by call to SetFpuFlags or temporary switch to C by this code:

with TSynFPUException.ForLibraryCode do
begin
  // call ffmpeg functions
end;

zen010101

Member

Registered: 2024-06-15

Posts: 171

Re: EInvalidOp crash on aarch64-linux when C library calls mORMot's export

Thank you Chaa, but TSynFPUException.ForLibraryCode cannot solve this issue.

  Our program structure is:

  // Main thread (FPC)
  TS_Init(apiKey, callback);     // Initialize SDK (libvct.so)
  TS_createTask(taskConfig);     // Create transcoding task
  // Main thread now waits for callback...
  while not Terminated do
    Sleep(10);

  The SDK (libvct.so) is a C library that internally uses GStreamer and FFmpeg. When we call TS_createTask(), the function returns immediately. The actual transcoding happens asynchronously inside libvct.so's internal threads.

  The crash occurs like this:

  Main Thread (FPC):              SDK Internal Threads:
    TS_Init()
    TS_createTask() ------>  libvct.so creates worker threads
    waiting...                    |
    waiting...                    v
    waiting...               GStreamer/FFmpeg processing
    waiting...                    |
    waiting...                    v
                             Thread "queue210:src" calls pow()
                             -> resolves to mORMot's p0w()
                             -> FPC ThreadVar not initialized
                             -> CRASH

  We cannot wrap our SDK calls with TSynFPUException.ForLibraryCode:

  with TSynFPUException.ForLibraryCode do begin
    TS_createTask(taskConfig);  // Returns immediately!
  end;
  // ForLibraryCode scope ends here, but crash happens
  // much later in a different thread we don't control

  The ForLibraryCode pattern only works when the C code executes synchronously in the calling thread. It cannot help when the C library spawns its own threads that later call pow().

  This is why we need a fix in mORMot itself - either a define like -dNOLIBCMATH or making the exported functions safe for uninitialized threads.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 15,383

Website

Re: EInvalidOp crash on aarch64-linux when C library calls mORMot's export

For a library like ffmpeg, I would never use static linking, but dynamic linking.