Error running docker container from ex\martin-doyle\07-HttpDockerORM

RaelB · 2023-10-28 20:00:13

Hi,
I can build the docker image from the martin-doyle\07-HttpDockerORM example.
However, when I run it, I get a list of AV's as below:

rael@User-PC:/mnt/d/fpcupdeluxe/ccr/mormot2/ex/07-Docker$ docker run -p 11111:11111 project07daemon
An unhandled exception occurred at $00007FC608B45C4E:
EAccessViolation: Access violation
  $00007FC608B45C4E

An unhandled exception occurred at $00007FC608B45B20:
EAccessViolation: Access violation
  $00007FC608B45B20

An unhandled exception occurred at $00007FC608B45B20:
EAccessViolation: Access violation
  $00007FC608B45B20

An unhandled exception occurred at $00007FC608B45B20:
EAccessViolation: Access violation
  $00007FC608B45B20

An unhandled exception occurred at $00007FC608B45B20:
EAccessViolation:
  $00007FC608B45B20

An unhandled exception occurred at $00007FC608B45C4E:
EAccessViolation:
  $00007FC608B45C4E

Not particularly helpful..

Any hints?
I installed lazarus and fpc with fpcupdeluxe, selecting the "stable" option, and using latest mormot.
Do I perhaps need to install some fixes?

I'm running Windows 10 x64, and building the app and docker image in WSL ubuntu 22.04.2

Thanks

Last edited by RaelB (2023-10-28 22:56:58)

RaelB · 2023-10-29 12:23:57

I did some investigation. Running Project02Server on an alpine image. First disabling all mormot code, and then enabling the units one by one.
mormot.core.base - no problem.
mormot.core.os - no problem.
mormot.core.log -> Problem occurs.
So looks like the problem lies in the InitializeUnit code there.

program Project02Server;

{$APPTYPE CONSOLE}

{$I mormot.defines.inc}
uses
  {$I mormot.uses.inc}
  SysUtils,
  mormot.core.base,
  mormot.core.os,
  mormot.core.log{,
  mormot.db.raw.sqlite3,
  mormot.orm.core,
  mormot.rest.http.server  }
  ;

Last edited by RaelB (2023-10-29 12:28:32)

mpv · 2023-10-29 14:16:40

mormot2 runs perfectly in container, we did it in TFB benchmark for a long time. Most likely problem because of alpine image, this image is based on truncated libc. Try with ubuntu or something with fulll libc

danielkuettner · 2023-10-29 16:38:54

Same behavior as under FreeBSD.
The AV occurs after unlocking a CriticalSection. I don't think it has to do with an old libc, but with using libthr (instead of libpthread under linux).
I guess it's not an Docker issue rather than an issue of the using image.

I'm not able to find a solution atm. The only thing I've found is the initialization of the CS will be called serveral time in right same thread (the thread of the initialization of the units.
Perhaps the libthr releases a CS when it will be initialized a second time. But it's just a guess.

RaelB · 2023-10-29 16:55:34

@mpv: Thanks for the info. I looked around a bit, and tried building the image with frolvlad/alpine-glibc - and it works without any issues!
There also is no need for the call to apk add.

So the dockerfile is

#FROM alpine:latest
FROM frolvlad/alpine-glibc:latest

#RUN apk add --no-cache libc6-compat sqlite-libs

WORKDIR /app

RUN mkdir data
RUN mkdir log

COPY /Project07Daemon /app/Project07Daemon

VOLUME ["/app/data","app/log"]

EXPOSE 11111

CMD ["/app/Project07Daemon", "--run"]

mpv · 2023-10-29 16:58:03

mORMot links to `libc`, but alpine image uses `musl` instead (this is not an old libc - this is another implementation of libc), so yes - this is not a Docker problem, but problem with running on alpine docker image.
See https://wiki.alpinelinux.org/wiki/Runni … c_programs about how to run on alpine if it is absolutely needed.
My opinion - do not use alpine at all - it creates too many problems, the gain of a few MB of disk space is not worth it.

JD · 2023-10-29 22:24:59

I have a mORMot 1 server in a Docker container built from Alpine Linux in production, and it runs without problems. However, my server's Docker build is based on the FreePascal example https://wiki.freepascal.org/Docker_Containerization

I later discovered and tried the mORMmot 2 based 07-HttpDockerORM example in a Docker container, and it worked for me. No AV issues at all.

However, I have to clarify that my development environment is Linux Mint and I adopted Docker because I did not want to create Windows & Mac versions of my mORMot server.

JD

Last edited by JD (2023-10-29 22:30:05)

ab · 2023-10-30 08:52:32

Some questions:

Do you use the mORMot memory manager?
Do you make some reallocations of large buffers?
The mORMot fpcx64mm.pas unit uses the mremap syscall for reallocating huge blocks. Perhaps it is a problem.

Or do you use the libc memory manager?

So first try to use the default memory manager for your project (slower but perhaps more compatible).
That is, comment the {$I mormot.uses.inc} line in your source.
Or try to define either FPCMM_NOMREMAP or OLDLINUXKERNEL in the mORMot 2 package options.

danielkuettner · 2023-10-30 17:14:28

I've testet with standard memory manager from fpc and with cmem.
No reallocations of large buffers of my own.
Now I've tested the two defines, but with no success.
(freebsd 13.2, fpc 3.2.2)

Last edited by danielkuettner (2023-10-30 17:15:54)

ab · 2023-10-30 17:38:25

What is the exact stack trace of the access violation?

danielkuettner · 2023-10-30 17:44:42

the segfault comes from thr_mutex.c when the CS will be unlocked a second time:

/*
* Check if the running thread is not the owner of the mutex.
*/
if (__predict_false(PMUTEX_OWNER_ID(m) != id)) <-- here
return (EPERM);

This comes from the second step of a for loop in mormot.core.rtti which calls

function TRttiCustomList.DoRegister(Info: PRttiInfo): TRttiCustom;

and here the finally RegisterSafe.UnLock;
calls in thr_mutex.c

mutex_unlock_common(struct pthread_mutex *m, bool cv, int *mtx_defer)

stack trace:
#0 mutex_unlock_common (m=0x90138e708, cv=false, mtx_defer=0x0) at /usr/src/lib/libthr/thread/thr_mutex.c:976
#1 0x0000000000431c19 in CTHREADS_$$_CLEAVECRITICALSECTION$formal ()
#2 0x00000000004ee651 in UNLOCK (this=...) at ../../mORMot2/src/core/mormot.core.os.pas:9003
#3 0x00000000004d9e59 in DOREGISTER (this=0x801637188, INFO=0xea9f78) at ../../mORMot2/src/core/mormot.core.rtti.pas:8629
#4 0x00000000004da677 in SETGLOBALCLASS (this=0x801637188, RTTICLASS=0xff6df0) at ../../mORMot2/src/core/mormot.core.rtti.pas:8750
#5 0x000000000094bffd in INITIALIZEUNIT () at ../../mORMot2/src/core/mormot.core.json.pas:11679
#6 0x000000000094c0b9 in MORMOT.CORE.JSON_$$_init$ () at ../../mORMot2/src/core/mormot.core.json.pas:11691
#7 0x0000000000421792 in fpc_initializeunits ()
#8 0x0000000000403cdf in main () at SOneSrv2.dpr:550

Last edited by danielkuettner (2023-10-30 18:45:20)

ab · 2023-10-30 18:51:33

@Daniel
Are you sure this is the same problem that the one faced with Docker ?

Please try with the new NODIRECTTHREADMANAGER conditional.
https://github.com/synopse/mORMot2/commit/e3bc8a8c
But I doubt it would fix the issue.

danielkuettner · 2023-10-30 20:11:52

Yes, I think that's the same issue like the docker one. But it's just my intension, because the av is released at the same time in the same unit initializations.

I've tested your new commit (thank's a lot for your investigation), but got the same segfault as without it:

mutex_unlock_common (m=0x901398708, cv=false, mtx_defer=0x0)
at /usr/src/lib/libthr/thread/thr_mutex.c:976
976 if (__predict_false(PMUTEX_OWNER_ID(m) != id))
(gdb) bt 10
#0 mutex_unlock_common (m=0x901398708, cv=false, mtx_defer=0x0)
at /usr/src/lib/libthr/thread/thr_mutex.c:976
#1 0x0000000000431c19 in CTHREADS_$$_CLEAVECRITICALSECTION$formal ()
#2 0x000000000042560a in SYSTEM_$$_LEAVECRITICALSECTION$POINTER ()
#3 0x00000000004d9ea3 in DOREGISTER (this=0x801637188, INFO=0xeb4b58)
at ../../mORMot2/src/core/mormot.core.rtti.pas:8629
#4 0x00000000004da717 in SETGLOBALCLASS (this=0x801637188, RTTICLASS=0x1001a30)
at ../../mORMot2/src/core/mormot.core.rtti.pas:8750
#5 0x000000000094ca5d in INITIALIZEUNIT () at ../../mORMot2/src/core/mormot.core.json.pas:11687
#6 0x000000000094cb19 in MORMOT.CORE.JSON_$$_init$ ()
at ../../mORMot2/src/core/mormot.core.json.pas:11699
#7 0x0000000000421792 in fpc_initializeunits ()
#8 0x0000000000403cdf in main () at SOneSrv2.dpr:550

ab · 2023-10-30 21:59:02

Is TRttiCustomList.Create called and RegisterSafe.Init properly called?

I don't understand why UnLock = LeaveCriticalSection estimates it is not running in the expected thread.

Try to replace the TOsLock type in RegisterSafe definition with TLightLock.
It should pass directly.
Performance should not be affected, because RTTI registration is likely to happen at startup, not at runtime from several threads, so it is very unlikely some thread contention do happen here.

danielkuettner · 2023-10-31 07:12:02

TLightLock works!
TOSLightLock not.

danielkuettner · 2023-10-31 08:18:31

Please give me some more time to find out next issues, because the main issue resists and just a first step of initialization of the units seems to work…

ab · 2023-10-31 13:29:17

TOSLightLock is used in a lot of places in the framework code.
It is supposed to be just a wrapper around a critical section, so we have to fix it for sure.

danielkuettner · 2023-10-31 13:43:19

Now with TLightLock my real app hangs at starting and I have to find out where.

ab · 2023-10-31 15:21:13

In fact, the lock should be reentrant.
So TLightLock is not enough.

Please try with https://github.com/synopse/mORMot2/commit/78bad373

danielkuettner · 2023-10-31 16:59:38

Thanks, with TLightLock app hangs again.

With TOSLightLock segfault again:

#0 mutex_unlock_common (m=0x90134b708, cv=false, mtx_defer=0x0)
at /usr/src/lib/libthr/thread/thr_mutex.c:976
#1 0x00000000004dcae3 in DOREGISTER (this=0x801de8370, INFO=0xeb7b68)
at ../../mORMot2/src/core/mormot.core.rtti.pas:8630
#2 0x00000000004dd357 in SETGLOBALCLASS (this=0x801de8370, RTTICLASS=0x1004fa0)
at ../../mORMot2/src/core/mormot.core.rtti.pas:8751
#3 0x000000000094f8bd in INITIALIZEUNIT ()
at ../../mORMot2/src/core/mormot.core.json.pas:11687
#4 0x000000000094f979 in MORMOT.CORE.JSON_$$_init$ ()
at ../../mORMot2/src/core/mormot.core.json.pas:11699
#5 0x0000000000421652 in fpc_initializeunits ()
#6 0x0000000000403b9f in main () at SOneSrv2.dpr:551

Registers:
rax 0x7fffffff 2147483647
rbx 0x90134b708 38674937608
rcx 0x801349008 34379960328
rdx 0x0 0
rsi 0x0 0
rdi 0x90134b708 38674937608
rbp 0x7fffffffd540 0x7fffffffd540
rsp 0x7fffffffd4e0 0x7fffffffd4e0
r8 0x6c6f6f42 1819242306
r9 0x80000 524288
r10 0x4a7b48 4881224
r11 0x0 0
r12 0x7fffffffd9c0 140737488345536
r13 0x0 0
r14 0x189be 100798
r15 0x801612000 34382880768
rip 0x800f06a33 0x800f06a33 <mutex_unlock_common+51>
eflags 0x10206 [ PF IF RF ]
cs 0x43 67
ss 0x3b 59
ds 0x3b 59
es 0x3b 59
fs 0x13 19
gs 0x1b 27
fs_base 0x800ef1120 34375405856

danielkuettner · 2023-10-31 17:14:52

with TLightLock app hangs when leaving InizializeUnit in mormot.core.log. After stepping over end; Debugger shows end; of FinalizeUnit.

Assembler show this:
0000000000421472 eb55 jmp 0x4214c9 <fpc_initializeunits+121>

0000000000421474 48c745f800000000 movq $0x0,-0x8(%rbp)
000000000042147C 488b45f8 mov -0x8(%rbp),%rax
0000000000421480 488d4001 lea 0x1(%rax),%rax
0000000000421484 488945f8 mov %rax,-0x8(%rbp)
0000000000421488 488b45f8 mov -0x8(%rbp),%rax
000000000042148C 48c1e004 shl $0x4,%rax
0000000000421490 488d15a99ba100 lea 0xa19ba9(%rip),%rdx # 0xe3b040 <INITFINAL>
0000000000421497 48833c0200 cmpq $0x0,(%rdx,%rax,1)
000000000042149C 7502 jne 0x4214a0 <fpc_initializeunits+80>
000000000042149E eb12 jmp 0x4214b2 <fpc_initializeunits+98>
00000000004214A0 488b45f8 mov -0x8(%rbp),%rax
00000000004214A4 48c1e004 shl $0x4,%rax
00000000004214A8 488d15919ba100 lea 0xa19b91(%rip),%rdx # 0xe3b040 <INITFINAL>
00000000004214AF ff1402 call *(%rdx,%rax,1)
00000000004214B2 488d05879ba100 lea 0xa19b87(%rip),%rax # 0xe3b040 <INITFINAL>
00000000004214B9 488b55f8 mov -0x8(%rbp),%rdx
00000000004214BD 48895008 mov %rdx,0x8(%rax)
00000000004214C1 483b5df8 cmp -0x8(%rbp),%rbx
00000000004214C5 7602 jbe 0x4214c9 <fpc_initializeunits+121>
00000000004214C7 ebb3 jmp 0x42147c <fpc_initializeunits+44>
00000000004214C9 488d0550aea100 lea 0xa1ae50(%rip),%rax # 0xe3c320 <TC_$SYSTEM_$$_INITPROC>
00000000004214D0 48833800 cmpq $0x0,(%rax)
00000000004214D4 7502 jne 0x4214d8 <fpc_initializeunits+136>
00000000004214D6 eb09 jmp 0x4214e1 <fpc_initializeunits+145>
00000000004214D8 488d0541aea100 lea 0xa1ae41(%rip),%rax # 0xe3c320 <TC_$SYSTEM_$$_INITPROC>
00000000004214DF ff10 call *(%rax)
00000000004214E1 488b5df0 mov -0x10(%rbp),%rbx
00000000004214E5 4889ec mov %rbp,%rsp
00000000004214E8 5d pop %rbp

DonAlfredo · 2023-10-31 17:52:50

This might help.
https://reviews.freebsd.org/rS350481

ab · 2023-10-31 17:53:50

TLightLock is not re-entrant so should not be used.
TOSLightLock is not re-entrant eigher so should not be used.

But TOSLock should work with no problem.

danielkuettner · 2023-10-31 18:32:20

@DonAlfredo
Thanks for this link. I've alfredy (joke) read about this issue. But the patch is fro 2019 and I've the R13.2 from Apr 2023 and should the backtrace not shown something of libc in such a case? But I will compare the patch with the current source files and perhaps that will help.

danielkuettner · 2023-11-01 07:39:59

@DonAlfredo
This patch is already included in R13.2 and it couldn't be an issue on linker side.

Thanks a lot for your hint!

ab · 2023-11-01 07:57:03

On BSD, the OSPTHREADSSTATIC conditional is set, so all pthread_* functions are resolved using static linking over the clib ('c'):

function pthread_mutex_init(mutex, attr: pointer): integer;
  cdecl external clib name 'pthread_mutex_init';

so it should use {$linklib c} as expected.
It works fine on MacOS.

Perhaps it is not good enough for FreeBSD.
In the FPC RTL, /freebsd/pthread.inc use this definition:

function  pthread_mutex_init (p:ppthread_mutex_t;o:ppthread_mutex_attr_t):cint; cdecl;external;

So there is no explicit link to clib.
And the same for /openbsd/pthread.inc (no explicit link to clib).
But for Darwin, the rtl defines in darwin/pthread.inc:

function  pthread_mutex_init (p:ppthread_mutex_t;o:ppthread_mutexattr_t):cint; cdecl;external 'c';

@danielkuettner
Perhaps try to remove the reference to clib in mormot.core.os.posix.inc and compile with:
https://github.com/synopse/mORMot2/commit/801a6b81
It may help solve the resolution problem.

danielkuettner · 2023-11-01 08:22:20

I've applied your commit, but
how can I remove the reference to clib in mormot.core.os.posix.inc? There are other functions (e.g. clock_gettime) needing clib.
Do I need cthreads anymore? We have disabled MemoryManager for testing.

ab · 2023-11-01 13:16:45

The cthreads unit is mandatory.

Just try with my latest commit.

It seems that our problem is only about pthread. Sometimes it is in clib, sometimes it is in pthreadlib...
The other functions like clock_gettime() are fine with explicit clib access.

danielkuettner · 2023-11-01 14:34:19

Ok, than no progress at all:

mutex_unlock_common (m=0x90134b708, cv=false, mtx_defer=0x0)
at /usr/src/lib/libthr/thread/thr_mutex.c:976
976 if (__predict_false(PMUTEX_OWNER_ID(m) != id))
(gdb) backtrace 10
#0 mutex_unlock_common (m=0x90134b708, cv=false, mtx_defer=0x0)
at /usr/src/lib/libthr/thread/thr_mutex.c:976
#1 0x0000000000434b49 in CTHREADS_$$_CLEAVECRITICALSECTION$formal ()
#2 0x00000000004254ca in SYSTEM_$$_LEAVECRITICALSECTION$POINTER ()
#3 0x00000000004dcae3 in DOREGISTER (this=0x801de8370, INFO=0xeb7b68)
at ../../mORMot2/src/core/mormot.core.rtti.pas:8630
#4 0x00000000004dd357 in SETGLOBALCLASS (this=0x801de8370,
RTTICLASS=0x1004fa0) at ../../mORMot2/src/core/mormot.core.rtti.pas:8751
#5 0x000000000094f8bd in INITIALIZEUNIT ()
at ../../mORMot2/src/core/mormot.core.json.pas:11687
#6 0x000000000094f979 in MORMOT.CORE.JSON_$$_init$ ()
at ../../mORMot2/src/core/mormot.core.json.pas:11699
#7 0x0000000000421652 in fpc_initializeunits ()
#8 0x0000000000403b9f in main () at SOneSrv2.dpr:551

If a do a breakpoint in thr_mutex.c for mutex_lock_common and mutex_unlock_common, than I can see, that this function will be called.
Are you sure, a confusion of libc and libthr functions exists?

danielkuettner · 2023-11-01 16:52:12

Please look at the stack trace from lazarus. Is #1 und #2 double or normal?
I'm not able to debug constructive (perhaps because of inlining).

#0 mutex_unlock_common(0x9013d1708, false, 0x0) at /usr/src/lib/libthr/thread/thr_mutex.c:976
#1 CTHREADS_$$_CLEAVECRITICALSECTION$formal at :0
#2 SYSTEM_$$_LEAVECRITICALSECTION$POINTER at :0
#3 DOREGISTER(0x801cef6a0, 0xe3d8d8) at ../../mORMot2/src/core/mormot.core.rtti.pas:8630
#4 SETGLOBALCLASS(0x801cef6a0, 0xf8af30) at ../../mORMot2/src/core/mormot.core.rtti.pas:8751
#5 INITIALIZEUNIT at ../../mORMot2/src/core/mormot.core.json.pas:11687
#6 MORMOT.CORE.JSON_$$_init$ at ../../mORMot2/src/core/mormot.core.json.pas:11699
#7 fpc_initializeunits at :0
#8 main at SOneSrv2.dpr:551

ab · 2023-11-02 08:53:18

Are you sure you did actually compile mormot.core.os.linux.inc ?
You need to modify mormot.core.os.pas or force "Compile Clean" in the mormot2 package manager.
Otherwise the include files are not recompiled by the IDE.

I am not certain there was a confusion between libc and libth, but our way to call pthread functions was not consistent with what cthreads did.

danielkuettner · 2023-11-02 09:07:26

Yes, I do always a clean & build under lazarus. But mormot.core.linux.inc? Do you mean mormot.core.os.posix.inc?

danielkuettner · 2023-11-07 10:45:54

I've tested and changed code in a very unprofessional way and maybe I'm lost, but I want to share my guess founded in mormon.core.rtti:

The address of the CS will be changed through line 8730 in TRttiCustomList.AddToPairs:

inc(Counts[Instance.Kind]);

After that the RegisterSafe.UnLock; in line 9002 goes with the wrong Pointer address in mutex_unlock_common and therefore it results in a segfault.

If I comment line 8730 the segfault is away and also my real app is running under FreeBSD 13.2 as under Win/Linux.

But I don't know, what's the meaning of line 9002 and the drawbacks of that.

Last edited by danielkuettner (2023-11-07 10:58:54)

ab · 2023-11-07 14:45:24

What is Instance.Kind value?
Is it rkUnknown?
I guess it is the case, but it should not for sure...

In this eventuality, https://github.com/synopse/mORMot2/commit/071b7abf should fix it.

Thanks A LOT for finding out this problem.

danielkuettner · 2023-11-07 15:42:06

Yes, with your last patch it works as expected.

Yes, it was rkUnknown.

mpv · 2023-11-07 15:55:47

Nice finding, for shure. Sometimes Pascal is unsafe....

ab · 2023-11-08 07:56:00

Do you know where this rkUnknown value came from?
Which type definition?

My wild guess is that this should not happen, every TypeInfo() should have its Kind field set in something realistic....

danielkuettner · 2023-11-08 10:41:39

In TRttiCustomList.DoRegister Info.KIND has rkBOOL.
But Instance (Result) will not set after GlobalClass.Create and given to AddToPairs(result, Info);

In AddToPairs I can't see from where Instance is getting his KIND (it's just rkUnknown) until inc(Counts[Instance.Kind]);

mORMot Open Source

#1 2023-10-28 20:00:13

Error running docker container from ex\martin-doyle\07-HttpDockerORM

#2 2023-10-29 12:23:57

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#3 2023-10-29 14:16:40

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#4 2023-10-29 16:38:54

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#5 2023-10-29 16:55:34

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#6 2023-10-29 16:58:03

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#7 2023-10-29 22:24:59

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#8 2023-10-30 08:52:32

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#9 2023-10-30 17:14:28

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#10 2023-10-30 17:38:25

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#11 2023-10-30 17:44:42

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#12 2023-10-30 18:51:33

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#13 2023-10-30 20:11:52

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#14 2023-10-30 21:59:02

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#15 2023-10-31 07:12:02

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#16 2023-10-31 08:18:31

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#17 2023-10-31 13:29:17

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#18 2023-10-31 13:43:19

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#19 2023-10-31 15:21:13

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#20 2023-10-31 16:59:38

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#21 2023-10-31 17:14:52

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#22 2023-10-31 17:52:50

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#23 2023-10-31 17:53:50

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#24 2023-10-31 18:32:20

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#25 2023-11-01 07:39:59

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#26 2023-11-01 07:57:03

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#27 2023-11-01 08:22:20

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#28 2023-11-01 13:16:45

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#29 2023-11-01 14:34:19

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#30 2023-11-01 16:52:12

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#31 2023-11-02 08:53:18

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#32 2023-11-02 09:07:26

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#33 2023-11-07 10:45:54

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#34 2023-11-07 14:45:24

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#35 2023-11-07 15:42:06

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#36 2023-11-07 15:55:47

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#37 2023-11-08 07:56:00

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

#38 2023-11-08 10:41:39

Re: Error running docker container from ex\martin-doyle\07-HttpDockerORM

Board footer