Splitting a big ERP written in Delphi 7 to microservices

mdbs99 · 2020-10-26 14:59:57

Hello everybody,

I'm going to split a big ERP to microservices. Those services will be consumed by ERP and a website written in Java.
My final goal is to write everything in FPC, but now the first services need to be written in Delphi 7, due the code is very "Windows like", using ADO for connection, some 3rd components, etc. The server also will be Windows, running IIS.

I have a good experience using FPC with FastCGI module on Windows+IIS, since its beginning. The newest version also have a lot of improvements, very different comparing early versions based on WebBroker... however, as those services need to be written in Delphi 7, I can't use FPC web framework.

I know that mORMot has everything that I need to write services in Delphi 7 (which will be rewritten in FPC in the future) but I would like to receive some ideas to use the best approach considering the environment and features that must have:

- Windows Server + IIS
- A/B deploy: service shouldn't stay off line, if we update a new version of it

Using FPC + FastCGI it's "easy" and I know the path... what about mORMot?

best regards, thanks.

ab · 2020-10-26 15:11:20

Define the services as interfaces, then publish them using mORMot.

On Windows, use the http.sys server. You can share the same port with IIS using proper URI registration.
This is a standard way of working with HTTP URIs on Windows - the same for DotNet WCF.
Our documentation also states about it.

About automatic updates, you may either
- deploy a stand-alone service, responsible for the updates
- self-replace the executable, making a copy of the current exe, then call it with some command line switches to do the rename - as with did with LiveMon agent

mdbs99 · 2020-10-27 13:27:46

I've never configureted the IIS + http.sys, so I would like a little more help, if you don't mind.

Well, I wrote the basics... now I have more information:
- First services will be used on local network, which means (for me) that I do not need to use IIS for them, but you see some advantage using it?
- The ERP code has a "classic" datamodule with a ADOConnection component, which is used all over the code. Should I initialize that connection in a specific place that is thread safe?

About automatic updates, I didn't work on it, but you remembered well. I will take a look on the sources.

macfly · 2020-10-27 13:55:49

ADO components are not thread safe.
Each thread must have its own connection.

I believe that it will not be a simple task to use ADO in conjunction with Mormot, due to the need to have this thread-connection control.

Following the post to see what ab says about it.

ab · 2020-10-27 14:21:39

If you can use OleDB instead of ADO (which is an overlayer on it), then you can use it with mORMot, since our SynOleDB unit has a thread-safe mechanism to maintain a connection per thread, and the mORMot HTTP server has a thread pool to reuse the threads.

mdbs99 · 2020-10-27 19:24:37

If I could change the main connection, won't be a problem using just mORMOt. Unfortunately, I can't.
I need to keep things working in ERP code, while the code is migrating for services—one code, multiple apps running.

Most of classes were coded following a good separation between GUI, rules and services. "Services" are classes that access external resources, like database (but using the global variable connection).

What I'm thinking to do—and I would like to read your opinions—is:
1. Implement a simple connection pool using ADO. Lock/create or get a connection avaible/Unlock.
2. Then, I'm gonna change those classes to use the pool instead the main connection. For the ERP perspective, it will be the same; but for the new mORMot services, it will be thread safe.
3. Repeat 1 and 2 for other classes/services.

Do you think that is a good strategy to rewrite the code?
What could be the better place—according to mORMot framework—to initialize this pool instance?

ab · 2020-10-27 20:16:47

Can't you use OleDB instead of ADO for the services code?
Which DB are you using?

What you define is called a "seam" in code refactoring.
Check https://www.slideshare.net/ArnaudBouche … conference

mdbs99 · 2020-10-27 20:33:17

I think I can use OleDB after the first refactor, due the deadline...
If I can code a pool for the same connection component, everything on ERP will continue working without MUCH changes, as the ERP will use 1 connection at all as always be.

I never used mORMot OleDB, so I don't know if I gonna have some issues (e.g. all queries will run smoothly?) and other things that I didn't think yet.
I'm using MSSQL 2008.

I will see the slides now...

Last edited by mdbs99 (2020-10-27 20:35:24)

mdbs99 · 2020-10-31 14:38:16

So, Arnaud, you asked which DB I'm using and I've answered above...

I can't use OleDB or any other connection type (in this first refactoring) because those services uses a lot of TADOQuery, which some has master/detail, events, and so on. I believe it will be simpler keeping ADOConnection, but I'm sure that I need to implements a connection pool for them, otherwise the new services/daemon won't have a good performance.

Do you have any more tips or advices about my thoughts on that architecture?

ab · 2020-10-31 15:18:34

In a first attempt, you can execute the services in a single background thread, and maintain a single connection in this thread.
If you don't have a lot of simultaneous users, it will work as expected.

Then try to put some data in cache, e.g. the most used queries, if you need more performance.

Later on, if you need to leverage the threads of the HTTP server thread pool, you can maintain an ADOConnection per thread, as we did for OleDB.

mdbs99 · 2020-10-31 22:26:43

I'll have simultaneous users and that is why I need multi thread.
Today we have a monolithic ERP using a single connect, but the main goal is to share the business rules on it on a website (written in Java).

The code has a TMainDataModule, which has a "singleton" MainDataModule variable. The ADOConnection is there. All other Forms and DataModule point to it.
Then, I thought in another possibility: What about using the MainDataModule as a threadvar?
- I don't need to implement my own pool
- the ERP code remains exactly it is (mostly)
- I "just" need to initialize/finalize the instance on each thread

What do you think?

mdbs99 · 2020-11-12 19:29:58

Guys,
I would like to share with you the solution that I've found, due my restrictions with the legacy code.

As I said before and here > Remote JSON REST Service - calling COM in Service fails, I'm using the MainDataModule as a threadvar plus overriding BeginCurrentThread/EndCurrentThread from TSQLRestServer to start it.

Event though I'm not using a connection pool (yet), the performance is good (~200ms average per request, executing multiple SQL (query and insert) in a MSSQL database and processing data to return for the client).

Javierus · 2020-11-12 21:05:15

What do you do from the begin/end thread? Open and close the connection?
Does the datamodule host all the app tables and business rules?

mdbs99 · 2020-11-12 21:12:53

@Javierus
Yes, I create the MainDataModule and open the connection.
No, this DataModule just have one ADOConnection and 2~3 ADOQuery for others thing that I'm not using on this service.
The business rules were coded (mostly) in classes, no DataModules. But all SQL/Scripts that are used by these classes are in ADOQuery among many DataModules, just for not write SQL as strings and use theirs master/detail features for reports, etc.

Last edited by mdbs99 (2020-11-12 21:17:02)

mdbs99 · 2020-12-08 15:09:24

From documentation:

16.4.2. Set up the Server factory:
"The code line above will register the TServiceCalculator class to implement the ICalculator service, with a single shared instance life time (specified via the sicShared parameter). An optional time out value can be specified, in order to automatically release a deprecated instance after some inactivity."

Is that option time out value which we're talking about all the time, to finish inactive threads?

ab · 2020-12-09 09:41:53

You could use the timeout to release your connections, indeed.
Check TServiceContainerServer.SessionTimeout and TServiceFactoryServer.InstanceTimeOut.
By default, the sicShared instances have a timeout of 30 minutes IIRC.

But note that the class instance will be running in several threads of the HTTP thread pool.

mORMot Open Source

#1 2020-10-26 14:59:57

Splitting a big ERP written in Delphi 7 to microservices

#2 2020-10-26 15:11:20

Re: Splitting a big ERP written in Delphi 7 to microservices

#3 2020-10-27 13:27:46

Re: Splitting a big ERP written in Delphi 7 to microservices

#4 2020-10-27 13:55:49

Re: Splitting a big ERP written in Delphi 7 to microservices

#5 2020-10-27 14:21:39

Re: Splitting a big ERP written in Delphi 7 to microservices

#6 2020-10-27 19:24:37

Re: Splitting a big ERP written in Delphi 7 to microservices

#7 2020-10-27 20:16:47

Re: Splitting a big ERP written in Delphi 7 to microservices

#8 2020-10-27 20:33:17

Re: Splitting a big ERP written in Delphi 7 to microservices

#9 2020-10-31 14:38:16

Re: Splitting a big ERP written in Delphi 7 to microservices

#10 2020-10-31 15:18:34

Re: Splitting a big ERP written in Delphi 7 to microservices

#11 2020-10-31 22:26:43

Re: Splitting a big ERP written in Delphi 7 to microservices

#12 2020-11-12 19:29:58

Re: Splitting a big ERP written in Delphi 7 to microservices

#13 2020-11-12 21:05:15

Re: Splitting a big ERP written in Delphi 7 to microservices

#14 2020-11-12 21:12:53

Re: Splitting a big ERP written in Delphi 7 to microservices

#15 2020-12-08 15:09:24

Re: Splitting a big ERP written in Delphi 7 to microservices

#16 2020-12-09 09:41:53

Re: Splitting a big ERP written in Delphi 7 to microservices

Board footer