Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

erick · 2020-01-17 21:16:15

Hi All,

I was facing the daunting task of converting a lot of mormot applications to Linux. The cost of Windows Server licenses and the updates were big factors, and also the freedom of not having to maintain more difficult Windows VMs.

However, there were a lot of libraries I have that aren't compatible with FreePascal's dialect let alone with Linux, and also my site is heavily invested in FreeBSD. So Linux conversion looked very challenging but not even the desired final result since I would prefer FreeBSD.

On a whim last night I tried Wine on FreeBSD (results on Linux should be same), and my mormot apps took less than an hour to convert to work. Actually, mormot itself worked right away, but I had to change a few other bits I use. They've been running great for 18 hours without incident.

For the uninitiated, Wine is a Win64 compatibility layer that loads Win64 applications on Posix operating systems and translates the Win64 apis to Posix apis allowing them to run under almost any Unix. All I needed to do was run pkg install wine64. You can run your Win64 console binary from a script or shell.

While native Linux support has great merit, the Wine solution means you can continue to use the Delphi compiler and debugger (on windows dev system), and many Win64 libraries, I used it with MariaDB's DLL to talk to MariaDB (MySQL-like database) running natively on FreeBSD. My other libraries talk to Office365, etc. and would have been a pain to convert.

Some libraries do not work with Wine, because they require Msvcr100.dll which you cannot instsall on a non-MS OS. But many work fine.

The hosted mormot app does not seem to be using many memory or cpu resources, not more than you would expect under Windows. You don't need to run X Windows, it works fine without it.

Just thought I'd share that in case other people had similar need for Delphi or 3rd party library support and were unable to move to Unix for those reasons. You probably can, for free, with Wine.

For the interested, there are two things that look like they are only stubbed in, and these may affect you but they didn't for my applications. NTLockFile looks stubbed, and also native language support, I think it defaults to English... but not sure.

Erick

ab · 2020-01-17 22:07:06

Hello,
Yes, Wine is an amazing peace of software.

For any serious work, I would go into FPC and Linux binaries (which can run on BSD AFAIR). We have mORMot Linux binaries server running more than 7 months now, with TBs of incoming/outgoing data, and hunderths of concurrent clients, with no stability nor performance problem - try to do this with a node or Java server

If you found out some fixes in the framework code to be support Wine, please make some pull requests.
Thanks for the feedback.

edwinsn · 2020-01-18 04:23:56

Hey erick, thanks for sharing, it's a very interesting and useful finding!

Over 2 years ago I had a similar experiment (https://synopse.info/forum/viewtopic.php?id=4263), and I've noticed significant improvements over the years from the media.

What I tried was on the Linux GUI, do you mean you can just install WINE from the FreeBSD command line and then run the Delphi-compiled, mORMot-powered console program from the FreeBSD console?

edwinsn · 2020-01-18 04:25:58

@ab,
It's perfect if one can port their apps to FPC which supports a lot of compilers, but as Erick has mentioned, with the WINE you can continue using Delphi and your 3rd party libraries, that's a huge benefit if you have time and resource constraints.

ab · 2020-01-18 10:14:49

My point was that if Wine support could be enhanced, e.g. by privileging some Windows API more stable with Wine - for instance the socket server should be more stable than the http.sys server on Wine - we could include that.

ab · 2020-01-18 10:15:29

My point was that if Wine support could be enhanced, e.g. by privileging some Windows API more stable with Wine - for instance the socket server should be more stable than the http.sys server on Wine - we could include that.

edwinsn · 2020-01-18 11:20:49

@ab, Yes, agreed! That'll be great if mORMot can include necessary changes, if any, in order to adapt to the WINE layer on Linux etc.

macfly · 2020-01-18 14:11:58

Thanks for sharing.

Currently for new projects I'm coding in Delphi and compiling in FPC.
Mormot is the foundation for creating cross-platform / compiler code.

But I have a legacy application where Wine would be a good option to run on Linux.

edwinsn · 2020-01-18 14:19:27

One question - I guess if you use WINE to deploy your Delphi program, you won't be able to debug on the target platform (FreeBSD or Linux)?

Last edited by edwinsn (2020-01-18 14:19:41)

erick · 2020-01-19 06:10:45

I had coded some huge apps in Delphi, and redoing in FreePascal would be challenging for a number of reasons. Even FreePascal/Win64 was a big hurdle and would be weeks or months of recoding by my estimate due to libraries and languages differences, and then more weeks to resolve Library differences on Linux, all this is extra time I don't have on a tight schedule.

So it becomes a question of how does one best spend their time.

Mormot + Wine may not be as efficient as native, but most apps don't need the killer performance of the microseconds that will be saved in not going through compatibility layer. The apps I'm referring to have hundreds of concurrent users not thousands. I haven't timed the variations, but I'm not convinced its any slower than native Windows server, FWIW. The question is not raw performance but reliability, and only time and testing will tell.

I will say: Delphi - for Windows - is very mature and is not undergoing frequent fundamental changes that break code compatibility. FP version hell is real, it can be a pain trying to match all the components for compatibility. I have struggled to get FP + mormot + zeos and Linux all to co-exist, I spent a whole day trying, retrying, etc., so this offers a faster entry. I hope in the long term that FP version stability will be resolved, but right now you cannot use FP/Current with mormot and zeos and linux, I believe, you have to back track a few releases. Correct me if I'm wrong about that.

Also, as a personal preference I'm more comfortable in the Delphi debugger on my workstation. And debugging in Delphi may not solve all your FP errors when you recompile and port to another OS. In contrast with compiling with Delph+Windows going to FP+Linux, Wine is like just like a slightly different version of windows, not unlike developing on Windows 10 and deploying on Windows Server. It's not as much of a leap.

The Wine developers request that we not detect and react to Wine, they'd prefer we tell them any problems and fix any API differences. There is an API call to detect Wine, but they request you only use that for gathering statistics or to debug your clients' problems.

As for debugging natively on posix, I'm not sure yet. You need to separately install win32 and win64 versions of Wine, because the universal delphi PAS server program is a Win32 program. And so far I have only installed Win64 support. I don't know if they install a device driver under Windows, haven't looked.

To answer Ab's question. I doubt the IIS emulation is ported, I only tried socket based calls as that seemed reasonable to me.

FWIW, the same mormot server applications also work under ReactOS iwhich shares some code base with Wine. I don't recommend it for production because it's still an alpha release and I believe it might only be Win32. But seriously, it's not unlike a JavaVM for your Win32 code, how cool. It may be useful for tiny 32 bit embedded computers - I ran ReactOS for 24 hours of mormot with no problems. YMMV.

One thing that doesn't work under Wine is the CData delphi extenion code. It looks like they have loader code which fails, possibly due to copy protection. You can't even load it and exit on the first line of your code. This is a shame because it would offer easy support for cloud app integration. I haven't invested a lot of time to debug it.

While your Win64 code cannot natively call Posix libraries unless you write some integration code, you can easily connect them by using HTTP/PHP/etc. or probably named pipes or similar. I have verified that you can do the HTTP/PHP integration for certain. Likewise, you can ps list or kill Win64 processes or restart them from Posix shell.

Here is the result of porting a somewhat complicated 12,000 line Delph/mormot/mariadb program. it took about an hour to get it to run under Wine64, it would have been faster but I had to rewrite the code to not spawn a new console, I had to remove and replace a DLL that requires msvcr100.dll which is cannot be installed on non-Microosft OS's, and I ran into the following permission bug that was a bit elusive.

While Mormot itself runs nicely as any unprivileged user - just as you would expect, I ran into one issue which took a while to determine but is easily addressable: the libmariadb.dll DLL file I was using for Win64 MariaDB connectivity only appears to work for userid root and not for unprivilegesd users. Exactly why I don't know, it may be resolvable at the Win64 level. Anyways, a quick FreeBSD solution to this problem is FreeBSD Jails, which a padded container-like feature that gives a read-only protected copy of the OS so the jail's root has almost no power, or I could run a single-purpose VM which is not quite as secure but could only expose itself. Under Linux there would be numerous virtualization options for sure.

I will be sure to test this mormot program extensively, Ibut was pleased with how little need to be changed.

BTW, if you have mastered FP, then all the power to you on native Linux, I'm not trying to dissuade anyone from native support, just demonstrating options available for those of us with special needs using the beauty and power of proven Open Source Software.

edwinsn · 2020-01-19 11:12:25

@erick,

Thanks for keep sharing, it's very helpful!

Re the msvcr100.dll issue, is it a license one? I mean, MS doesn't allow installing msvcr100.dll on non-MS OSes, right?

BTW, I assume Linux is much much more popular then FreeBSD nowadays, I'm just out of curiosity - why do you prefer the latter?

erick · 2020-01-19 12:51:47

edwinsn wrote:

@erick,
Thanks for keep sharing, it's very helpful!
Re the msvcr100.dll issue, is it a license one? I mean, MS doesn't allow installing msvcr100.dll on non-MS OSes, right?
BTW, I assume Linux is much much more popular then FreeBSD nowadays, I'm just out of curiosity - why do you prefer the latter?

msvcr100.dll is used for ... I believe... MS Visual C compiled DLLs in multithreaded environments. I think you need to run the MS installer for msvcr100.dll, and I believe it checks compatibility with 'genuine' licensed Windows. I didn't explore further, because I found alternate libraries which didn't require msvcr100.dll, which was a more attractive solution to me.

Since you are curious about FreeBSD. Please take all the following with a grain of salt, I don't speak for FreeBSD but I have used it since Version 1.0.

The choice is really six of the pros and a half dozen of the cons. A talented computing professional can work as easily in Linux as FreeBSD. The real answer is history, we ran FreeBSD before Linux was reliable and popular, and there is some inertia and a large infrastucture with little reason to change, but also other things I will describe below.

In web servers in 2019, there were 1.2 billion sites and many more private servers. Linux had about a 70% Web market share, FreeBSD had a 1% market share, but it's still in the order of Millions of installations when you realize all the routers, servers, VMs etc. But that's not a complete picture, Linux is fragmented among its adherents so each distro's share is less than the total 70%. I don't know the breakdown between Linuxes today, it's not possible to tell from Web site visits.

Having set up numerous Linux systems, there is an occasional bit of frustration that programs you want are not always readily available for your distribution. They have to be ported due to differences in the distributions. Sometimes you need a RedHat, sometimes Ubuntu, sometimes Raspberian, etc., and sometimes specific versions. And sometimes you end up running multiple versions of the OS for different applications because you gave up after hours or days of trying to port it.

FreeBSD offers a few advantages (and a few disadvantages). It offers ZFS as the primary stock filesystem and OS jails which run read-only virtualized versions of FreeBSD where root is not really all-powerful, and BHyve virtuatlizaiton of any OS, and there are certain other features that are available because of its licensing that don't allow stock in GNU licensed products due to copyleft. But where it is also nice is that everyone running a particular version is compatible with all the packages - you don't have the same splinterring as in the linux community.

There is no perfect OS for compatibility. But with virtualization, who cares, we can run them all.

I find some linux distros are bleeding edge, with alpha quality drivers frequently showing up for bleeding edge hardware - so they run on the widest selection of hardware, but at the cost of reliability. Others linuxes are very conservative, like RHEL (I believe), which presents other issues as we couldn't get it to work with some packages that are too new. FreeBSD has a slightly different balance, not running on the most esoteric hardware, not having the most packages available, but reasonable compromises on both.

If you hand pick your hardware and sofware (which you should for any server), you will probably find no difference between the BSDs and the Linuxes for speed or reliabilty. And if you are a complicated site, you probably have several different Linux distros virtualized for some compatibility, and indeed, we run a few linux VMs, its just easier that way.

Mind you, we use a common file server and the virtualizaiton lets us use one firewall ruleset for all VMs, which makes it simpler than rememberring each OS's firewall rule languages.

Don't believe speed tests, people are always comparing the two and each release from both FreeBSD and Linux are faster and better than the last. The tiny margin one has over the other is short-lived. And I believe the friendly competition is good for the community to strive to be better and constantly innovate, but in reality they share so much from the development community and cross pollination.

So FreeBSD is not a worse choice, and not a better choice, just a reasonable choice. There is no one Linux which can serve all software needs, or we would all migrate to it and that would be the end of all other distros and FreeBSD.

Erick

edwinsn · 2020-01-19 17:59:55

Thanks Erick

erick · 2020-01-20 00:23:47

One more thing of interest: some people may be running Windows VMs to host the Delphi Mormot applications. At least I am.

I addition to the Windows licensing cost, you would wonder about the physical resources required with Wine versus Windows and whether the API conversion layer slows down your system. It's a reasonable assumption, right?

I conducted these tests at home, running on a Virtual Box system hosted under OS/X, latest version, with SSD and 16 GB of RAM. There was no swapping as I made the VMs sufficiently large.

I set up a Windows 10 VM with 10G of RAM, 2 cores and sufficient disk space for Windows, a mormot App and MySQL.

Then I set up a FreeBSD VM with 2 GB of RAM, 2 cores, Wine, the identical mormot app binary and MySQL.

For my tests I moved both VMs off screen, so there would be no video updates interfering with the performance (that makes a difference), and there were no flashing cursors on the VM.

Results:

Windows: 33% of CPU, 138 MB RAM
FreeBSD/Wine: <3% of CPU, 113 MB RAM

So while memory usage was near identical, Wine was 10 times more efficient with CPU than hosting a Windows 10 VM when idle. Even with light usage, the numbers did not change. I ran the tests for a half hour, just in case Windows was downloading updates or something, but no changes happened.

I still need to determine the resiliancy of the image, but so far everything is looking good.

Erick

ab · 2020-01-20 09:13:36

All this does perfectly make sense!

Thanks Erick for the feedback.

Edit/additional question: Did you encounter any problem with time measurement under Wine, in the logs?
https://golang.org/src/runtime/os_windows.go seems to detect some Wine bugs about QueryPerformanceCounter and GetSystemTimeAsFileTime APIs, which may be lacking or incorrect ?

edwinsn · 2020-01-20 09:31:30

Hey Erick,

The result is very much impressive! Do you have a link to any good article that you can suggest, mostly about two things:
- Install WINE from Linux console that's hosted in a remote VPS
- Get a mORMot app run by WINE.

ab · 2020-01-20 11:16:35

I have added Wine detection, especially to have the Wine revision in the logs, which could be very handy.
See https://synopse.info/fossil/info/3a602921ae
And also added a warning in the logs if the application tries to use http.sys (httpapi) which is not correctly implemented in Wine (the sockets server is to be used).
So there is no adaptation - as Wine developers expect - but just detection for proper debugging.

What would be interesting is to run the TestSQL3 regression tests under Wine.

erick · 2020-01-20 22:30:15

edwinsn wrote:

Hey Erick,
The result is very much impressive! Do you have a link to any good article that you can suggest, mostly about two things:
- Install WINE from Linux console that's hosted in a remote VPS
- Get a mORMot app run by WINE.

Hi, I'll try to answer the questions posed.

VPS? Should work, I can't see why it wouldn't. In fact, I installed it on my development VM over a VPN from home on the weekend, so that's effectively like a VPS.

Wine DOES require root permissions to install, normally, because it puts shared libraries, etc. and the wine64 loader program into /usr/local. So if you have root permissions of your computer or VM, you can install it. For mormot applications, you do not normally need X windows to be installed or running. So console root shell is generally sufficient. Current Wine binaries are readily available, you don't need to MAKE the installation, and disk requirements are minor. Yum or apt can be used.

The applications are all unprivileged, running as the current effective user. It does not emulate a Windows computer or the Intel processor, like VirtualBox or similar do which use VT instructions and ring3 - your code runs natively on the Intel/AMD cpu without ring changes or device drivers. Wine simply translates the Win64 API functions to Posix functions as it executes.

I say "simply" because the system is not managed by device drivers and complicated ring transitions - I'm coming from a lifetime of writing deivce drivers which would do that. Inside the wine DLLs there is certainly is a good amount of cleverness to emulate the registry, map in 'virtual drives' C:\ and Z:\, etc. But these are all predictable user-mode translations. Look in ~/.wine for the virtual registry, virtual drive mappings through symbolic links, etc. Can you imagine, a worderful ANSI based registry you can parse with grep!

Many 'changes' to your environment, such as hiding error/warning messages spit out by the wine64 can be managed simply with environment variable switches. However, I would say changes are not required for many or most mormot programs unless you want to hide the few warnings at startup and shutdown time.

The regression test fails on a few matters, (like HTTPD.SYS), but also because Unix is more cautious about somethings, like allowing TCP ports to be opened below 1024, etc. that Windows allows. Another issue not handled as well is memory mapped files. If you do them, you will want to test and read up on the subject.

I can vouch that Zeos and sqlite3 databases worked perfectly for me. But I don't know that Window's file locking is translated to the posix level, I kind of think it is just emulated at the process level and applicable to only the current process.

Earlier I showed that wine64 is 10 times less CPU intensive than running Windows 10 VM. I should note that I did not tune the systems or prune any processes in the installation, so Win10 was running all the annoying little services enabled by MS by default, just as my FreeBSD VM was.

The fact that you aren't running any extra services probably does reduce your attack space compared to having a whole Microsoft OS VM loaded.

Something that is available, but I haven't explored, is that you can link Wine's dlls directly into your binary, and thus ship a read-to-use Linux, etc. programs that I BELIEVE does not require Wine to be root-installed. I don't need this functionality so I'm not going there, but maybe others with limited shell accounts might want to investigate further.

edwinsn · 2020-01-21 04:16:44

Hey Erick, thank you again for the very insightful and helpful info!
So in order for this kind of web server programs to run under WINE, all you need is a Delphi-compiled console mORMot program that that doesn't have a VCL GUI.

Last edited by edwinsn (2020-01-21 04:18:39)

erick · 2020-01-21 04:30:20

Certainly no VCL or FMX. Also, I'm not guaranteeing it.

It also cannot have createasconsole() as some of the examples have. And there are a few other functions that might not work. And some external DLLs might not load. And I don't know about any ActiveX related functionality. Also you have to observe rules of the Posix system, such as not using protected ports, etc.

So you will have to test it yourself, but I'm saying it's doable with only minor effort for a large number of mormot programs. YMMV.

Erick

macfly · 2020-01-21 13:13:38

Just for curiosity. Is anyone using (or has used) the Windows Nano Server?

Before or after changes in the form of distribution.

It is quoted in the Mormot documentation, but I have never seen comments about him on the forum.

erick · 2020-01-22 00:27:01

I read that the Nano server was practically killed by the change to only support containers back in 2017.
Google death nano server for details.

Erick

ab · 2020-01-22 01:15:55

Yes, I could just erase this part of the documentation.
Especially since we support Linux, and an Alpine Linux container is so small and stable... and BSD could also be small & beautiful.
We were very hopeful about Windows-Nano, but it was at a time where we supported only Delphi and Windows...
I guess BSD+Wine or Linux+Wine could be a better solution if you need to stick to the Windows API, as Erick stated!

pvn0 · 2020-01-22 12:31:45

I wanted to use Alpine Linux but it's really annoying it uses musl libc when most popular distros use glibc makes it more difficult to move programs around. I was really glad to see fpcupdeluxe adding support for Alpine (and other musl distros) some time ago.

Last edited by pvn0 (2020-01-22 12:31:58)

macfly · 2020-01-22 14:06:55

I saw that it was a bad decision by Microsoft. So I asked if anyone had used it before the change in the form of distribution.

Let's go back to the topic...

AOG · 2020-01-22 20:38:43

Just for fun ... good mORMot results on Alpine Linux !!

ab · 2020-01-22 21:03:19

No static linking of SQLite3 ?

pvn0 · 2020-01-22 21:10:00

ab wrote:

No static linking of SQLite3 ?

but then the obj files need to be recompiled I think

ab · 2020-01-22 21:11:37

Because they were compiled with glibc, not muslibc?
IIRC we included all needed .o .a files...

AOG · 2020-01-23 09:01:28

Static ... also just for the fun of it ... ;-)

Will do a pull for the changes needed for static.

Leslie7 · 2020-01-27 10:08:34

Faced with the same issues I came to conclusion that on the long run the best solution is to create a background server with delphi running under wine which the main linux server calls to do certain things it cannot do itself. Only those libraries go to the background server which cannot be replaced with fpc/lazarus equivalent. Everything else runs native under linux. Sometimes the background server may need to call some interfaces on the main server to do its job.

The problematic area with running a command line server under wine for me is making it to start after reboot. I could not find a way to autostart more then one server. But my linux knowledge is very limited. I thought maybe it should be started by the main server instead configuring Linux to do it, but could not get it working either. Did not try very long though.

Last edited by Leslie7 (2020-01-27 10:16:14)

edwinsn · 2020-01-27 11:57:05

3 days after Erick's post, WINE 5.0's released: https://www.winehq.org/news/2020012101

mORMot Open Source

#1 2020-01-17 21:16:15

Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#2 2020-01-17 22:07:06

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#3 2020-01-18 04:23:56

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#4 2020-01-18 04:25:58

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#5 2020-01-18 10:14:49

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#6 2020-01-18 10:15:29

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#7 2020-01-18 11:20:49

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#8 2020-01-18 14:11:58

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#9 2020-01-18 14:19:27

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#10 2020-01-19 06:10:45

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#11 2020-01-19 11:12:25

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#12 2020-01-19 12:51:47

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#13 2020-01-19 17:59:55

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#14 2020-01-20 00:23:47

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#15 2020-01-20 09:13:36

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#16 2020-01-20 09:31:30

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#17 2020-01-20 11:16:35

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#18 2020-01-20 22:30:15

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#19 2020-01-21 04:16:44

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#20 2020-01-21 04:30:20

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#21 2020-01-21 13:13:38

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#22 2020-01-22 00:27:01

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#23 2020-01-22 01:15:55

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#24 2020-01-22 12:31:45

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#25 2020-01-22 14:06:55

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#26 2020-01-22 20:38:43

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#27 2020-01-22 21:03:19

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#28 2020-01-22 21:10:00

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#29 2020-01-22 21:11:37

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#30 2020-01-23 09:01:28

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#31 2020-01-27 10:08:34

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

#32 2020-01-27 11:57:05

Re: Using Mormot on FreeBSD, Linux, etc. with almost all your windows libs

Board footer