Posts by squirrel

squirrel · mORMot 2

Thank you so much for this @ab!

Besides being very very useful, it also makes the code more readable and understandable, especially when someone unfamiliar with mORMot needs to work on/review our code.

High-level wrappers like these makes it possible for the power of mORMot to be fully used without needing to know the technical details of everything that is possible, but undiscoverable in this great framework.

squirrel · mORMot 2

Thanks @ab. Looks like it is working now.

squirrel · mORMot 2

If there is a way to get OnBeforeURI to be triggered in this scenario, then I can do a redirect or handle it in many ways.

squirrel · mORMot 2

If it is possible, that will be wonderful! Thanks @ab!

squirrel · mORMot 2

Thanks to @htits2008 for pointing me in the right direction. For anybody else affected by the same issue, here is a procedure to fix it. Just remember to add all new methods where you call it, otherwise they will still be affected by it. Maybe there is a way to get a list of all the method based functions (not the interface functions) and step through them at create time so that this procedure can be called without parameters? Will check later when more time is available:

procedure FixURIRoutingMess(const HttpServer: TRestHttpServer; const methods: array of string);
var
  I, R: Integer;
  J: TUriRouterMethod;
  tmpURI: string;
begin
  for I := 0 to Length(methods) -1 do
    for R := 0 to HttpServer.RestServerCount -1 do
    begin
      tmpURI := '/' + HttpServer.RestServer[R].Model.Root + '/' + methods[I];
      for J := urmGet to urmHead do
        HttpServer.Route.Rewrite(J, tmpURI + '/', J, tmpURI);
    end;
end;

This can be called something like this (based on the pastee sample above):

  FixURIRoutingMess(aHttpsServer, ['method1']);

squirrel · mORMot 2

Unfortunately this doesn’t only affect my root function, but All my method functions. It is breaking bookmarked links at the clients and 3rd party integrators to the extent that upgrading to mormot 2 is now blocked.

squirrel · mORMot 2

Is there any way to get OnBeforeURI to trigger in this scenario in mORMot 2 like it does in 1?

squirrel · mORMot 2

If it is possible to get the OnBeforeURI to trigger for when the uri has a trailing / then I can do a redirect or handle it somehow.

squirrel · mORMot 2

I'm using Delphi 10.3.

example-03:
in u_SharedTypes, the following line causes [dcc32 Error] u_SharedTypes.pas(75): E2034 Too many actual parameters
pmCheckedFileName^ := TPath.Combine(dirName, fileName, False)
Removing the ,false parameter will cause it to compile.

I hope there is a way to restore the mORMot1 uri parsing that allows the trailing /, since I am porting projects to mORMot 2 that depends on it being consistent.

squirrel · mORMot 2

The example-03 does not compile, but I've created my own tiny example dpr here which shows the difference between mORMot 1 and 2 and can be used to highlight/test the issue:
https://paste.ee/p/tKeqf

squirrel · mORMot 2

I notice a difference between how urls are parsed between mORMot 1 and 2 and was wondering if there is a setting to get the original behaviour back. Sample code:

  TMyRestServer = class(TSQLRestServerFullMemory)
  published
    procedure Method1(Ctxt: TSQLRestServerURIContext);
  end;
...
procedure TMyRestServer.Method1(Ctxt: TSQLRestServerURIContext);
begin
  Ctxt.Returns('{"value":"hello"}');
end;

When using mORMot1, accessing all these urls will give the same response:
https://127.0.0.1:8084/test/Method1
https://127.0.0.1:8084/test/Method1/
https://127.0.0.1:8084/test/Method1/someval

{"value":"hello"}

But with mORMot2, all of the above works except for https://127.0.0.1:8084/test/Method1/ which returns:
{"errorCode":400,"errorText":"Invalid URI"}

Is this the expected behaviour?

squirrel · mORMot 2

Thanks ab.

I did look at the long work sample before posting and it did look promissing. But it looks like that would require clients redevelop their existing solutions to use websockets.

Always difficult justifying breaking backwards compatibility to 3rd parties. Also not too sure how to modify almost 200 rest interfaces and their clients to fit into that workflow.

squirrel · mORMot 2

Increasing the thread pool size is the quick solution, yes. But that misses the point. The point I was trying to make, is that there are different uses for rest calls on the same system. Some are quick and high priority, such as logging in and editing records. It is reasonable to expect them to complete in a few ms, and not consume 32 threads. However, there are other types of functionality that takes longer, but are lower priority, such as running complicated reports.

The idea is to be able to put "slow" type tasks in their own queue and not allow them to bring the whole rest server to a halt. If the thread pool size is simply increased to 32, then running 32 slow reports (or 1 slow report being used by 40 users) will effectively prevent other users from completing higher priority tasks such as logging in and editing records until those slow tasks have been completed.

If there is a functionality to create "tasks" for those requests, they can be placed in a queue or something to manage them, releasing them from the rest server's thread pool until they complete and perform a callback to instruct the rest server to complete the request.

squirrel · mORMot 2

I have interface based services running on TRestHttpServer, which provide reports to third parties. One of the reports can be very slow and can take 15 or 20 seconds to complete. Is it possible to offload such a rest request to its own task without blocking the other worker threads or the http server?

Not sure I'm explaining it very clearly. To test for options, I've changed this sample \mORMot2\ex\ThirdPartyDemos\martin-doyle\04-InterfacedBasedServices and added the following function to the TExampleService rest interface:

function TExampleService.Time(var WaitSeconds: Integer): RawJson;
var
  StartTime: TDateTime;
begin
  StartTime := now;
  Sleep(WaitSeconds*1000);
  Result := '{"startTime": "' + DateToISO8601(StartTime) + '", "endTime": "' + DateToISO8601(now) + '"}';
end;

Then reduced the threadpoolcount from 4 to 1. The effect is, as expected, that several requests to this function are serialized, so running:

curl http://localhost:11111/root/example/Time?WaitSeconds=10

at the same time in different windows produce the following results:

{"result":[10,{"startTime": "2023-09-28T12:33:55.415Z", "endTime": "2023-09-28T12:34:05.420Z"}]}
{"result":[10,{"startTime": "2023-09-28T12:34:05.421Z", "endTime": "2023-09-28T12:34:15.430Z"}]}

This has the effect that the requests are executed one after the other, blocking new requests. My initial thought was that mORMot's new async http server would be ideal for this. But since these are rest requests already used by 3rd parties, I can not change that to websockets at this stage.

squirrel · mORMot 2

Is there something similar to the CurrentServiceContext from mORMot 1 in mORMot2?

squirrel · mORMot 1

Hi @ab
Not sure where to look, has OAuth 2.0 support been added to mormot 2?

squirrel · mORMot 1

Has anybody had any success implimenting this oauth2.0 into their project?

squirrel · mORMot 2

@mpv thanks that is a good suggestion. I'm a huge fan of postgresql. Unfortunately Mysql is mandated for this project.

Off topic: I really miss PGAdmin3. The web based tools always feel like swimming in syrup. I know there are lots of db tools for pg that works great, but typing in pgadmin3 always brought a smile to my face

squirrel · mORMot 2

Thanks @ab. I'll take a look.

Re the parameter binding, is this what you are referring to? https://dev.mysql.com/doc/c-api/5.6/en/ … param.html

squirrel · mORMot 2

I'm planning to start migrating one of my larger projects to mORMot 2 later this year. Since that will be a fairly big change anyway, the plan is to put the project on a diet and reduce the number of third party libraries it uses.

While looking at the tfb benchmarks for mORMot (seriously seriously impressive - thanks guys!!!), I was wondering if it would be possible to do something similar to mormot.db.raw.postgres and create mormot.db.raw.mysql to directly use libmysql.dll? That would remove the need for using firedac (and its quirks) in order to use libmysql.

Does anybody with more knowledge of libmysql know how do-able this would be?

squirrel · mORMot 1

Is it possible to get the number of rows with ISQLDBROWS in Mormot 1? I need to know if no rows were returned. While it is possible to set a local boolean variable first to false and then to true during each ISQLDBROWS.Step, it will add to an already complex code base.

Is there something similar to TSQLDBStatement.TotalRowsRetrieved that can be used after the code has stepped through the data to determine if an empty set has been returned?

Something along the lines of:

var 
  data: ISQLDBROWS;
begin
  data := Connection.Execute('select * from table', []);
  while data.step do
    ...

  if data.TotalRowsRetrieved = 0 then
    HandleNoData;

EDIT: Please ignore my idiocy, I completely forgot about being able to just do: data.Instance.TotalRowsRetrieved

squirrel · mORMot 1

Please allow me an uneducated question (this is far outside my expertise):
Would building these same benchmarks on both Fpc and Delphi compilers and running them on the same hardware provide any meaningful comparison between the compilers as well or would the current tests not provide relevant data?

squirrel · mORMot 1

Will the InitArrayFrom still provide the same structure / result when it is not an array? I thought that would only apply to dynamic array types.

squirrel · mORMot 1

It doesnt look as if the InitArray* functions do the same

squirrel · mORMot 1

Thanks TOrmTableWritable works perfectly. I've been using windows grep for most of the day searching for the replacements. Not sure how I missed this one. Looks like my list is now starting to become useful. I should be able to start migrating as soon as I find out about TDocVariantData.InitFromTypeInfo

squirrel · mORMot 1

What is the mormot 2 class for TSQLTableWritable called?

squirrel · mORMot 1

Is there a document available which lists which units different declarations have moved to? For example TDocVariantData moved from SynCommons to mormot.core.variants etc?

I am trying to find the location for TDocVariantData.InitFromTypeInfo and also busy making a list of common mormot types that I use a lot and where they are now to make migrating existing projects easier.

squirrel · mORMot 1

Thank you so much!

squirrel · mORMot 1

Does anybody still have copies of these gists available? I need a mormot server with oauth authentication and was hoping this would be something to build on. But the gists seem to be all gone already.

squirrel · mORMot 1

Our thoughts and prayers are with you and the Ukraine people. This war should never have started. We hope it ends quickly.

squirrel · mORMot 1

Hi @ab
In your blog post about mORMot 2 performance, you mention that "The official release of mORMot 2 is around the edge". We are very curious (and excited) for that day. Is there a specific date or feature parity that must be reached or are we getting close?

squirrel · mORMot 1

Thanks Arnaud. This is very exciting and must have taken you a lòt of time!
So much appreciated. It is a very useful expansion of the framework.
Can't wait to get my hands on the examples and start playing with this.

squirrel · mORMot 1

Yes, that is probably it. Maybe its a browser issue. I wont worry too much about that now.

squirrel · mORMot 1

32 threads

Maybe this is a browser issue and not a framework issue. It happens when I test using two tabs of the same browser, but not when using two different browsers at the same time, such as chrome and firefox.

squirrel · mORMot 1

I have done something similar to the documentation, but it looks as if all requests are then executed sequentially. I created a method that executes the functions like this

  if Services['Calculator'].Get(svr) then
    if SameText(req.FunctionName, 'Add') then
      Ctxt.Returns(svr.Add(1, 2))

To come to that conclusion, I did something like this in Add:

...
  StartTime := now;
  Sleep(5000);
  StopTime := now;
  Result := '{"function": "Add", "StartTime": "' + DateTimeToStr(StartTime) + '", "StopTime": "' + DateTimeToStr(StopTime) + '"}';
..

When I open 2 browsers and call the above method at almost the same time, the results indicate that they are executed one after the other:

browser1: {"function": "Add", "StartTime": "2022/05/12 11:26:56", "StopTime": "2022/05/12 11:27:01"}
browser2: {"function": "Add", "StartTime": "2022/05/12 11:27:01", "StopTime": "2022/05/12 11:27:06"}

Is this the expected behaviour, or am I doing something wrong? The interface was declared with sicShared, but the same effect is seen when using sicPerThread or sicSingle .

squirrel · mORMot 1

I need to temporarily put something in place of a Datasnap server that third parties integrated to. So what is important, is that I maintain exactly the same uri formats. Unfortunately that means that third party clients over which I have no control, need to be able to make POST requests such as /root/Calculator/Add/1/2 which I need to translate to the registered interface functions, ie to ICalculator.Add(1, 2) and execute it.

squirrel · mORMot 1

How would I then find and execute the correct function on that interface with the correct parameters? I assume I'll have to search for the methodindex from the interfacefactory and then step through and assign the declared parameters?
Have someone done something like this before?

Since ExecuteCommand is private, how would the function be called after the parameters are assigned?

squirrel · mORMot 1

Is it currently possible to do for example

GET 'customers/{CustomerId}/invoices/{InvoiceNumber}'
or
POST /root/Calculator/Add/1/2

using interface based services?

squirrel · mORMot 1

TSQLRestServerFullMemory

squirrel · mORMot 1

It looks as if it always thinks the binary data is json, so maybe that is used as a fallback when it can't determine the data type? Since these are partial files, not full files (which would be too big - up to 1 gb), I assume determining the content type will not succeed. How can I provide / override / force the content type when serving binary data?

Providing the full path of the original file does not resolve the issue - is still sends the content-type as json.

squirrel · mORMot 1

I am attempting to correctly set the content-type when sending parts of binary files using code like this:

        AddToCSV('Accept-Ranges: bytes', Ctxt.Call.OutHead, #13#10);
        AddToCSV('Content-Range: bytes '+ inttostr(RangeStart) +'-'+inttostr(RangeEnd)+'/'+inttostr(SrcFileSize), Ctxt.Call.OutHead, #13#10);
        AddToCSV(GetMimeContentTypeHeader('',SrcFileName), Ctxt.Call.OutHead, #13#10);
        Ctxt.ReturnBlob(blob, HTTP_PARTIALCONTENT);

In the procedure, the content-type is correctly set as video/H264 in the Ctxt.Call.OutHead, but when the header reaches the client, it is content-type: application/json; charset=UTF-8

Any suggestions where this might be overridden? I have tried stepping through ReturnBlob, but when it returns to my proc, the header still looks correct, so I am not sure where it is happening.

Using v1.18

squirrel · mORMot 1

Hi ab

Streaming the file directly from the server is great. I wish this was possible with mORMot 1. If I may get back to the second half of dexter's question - saving of the file by the client. Is there a way for the client (ie TWinHTTP.Request) to stream the bytes received from the server directly to file/disk without caching it in memory first?

squirrel · mORMot 1

That would work, I'll just have to consider that the functions in the dll will need db access and should not be able to be used by other unauthenticated programs/users

squirrel · mORMot 1

Would the opposite of this be possible?

I have a generic mormot server that is used for authentication and to perform some calculations and database actions. This runs offline directy at customers' sites. For some customers, some of the interface functions must be replaced to do calculations differently.

Since I do not want to recompile the exe for every customer request, I'm hoping to move only those functions to a dll and have the mormot server use the custom versions of the functions in the dll if they are present.

From my understanding of TSQLRestServer.ExportServer, the expectation is that the entire mormot server will live inside the dll. Is that correct? I hope to keep the rest server inside my main exe (since that is its only reason to exist) and just extend it with additional or replacement interface functions from the dll. Would that be possible?

squirrel · mORMot 1

Yes, extracted the latest from that release into the static folder.

squirrel · mORMot 1

It is caused by mormot.db.raw.sqlite3.static

squirrel · mORMot 1

Unfortunately it does not specifiy a specific unit, just mormot2tests.dpr being open.
I assume that still having the mormot 1 paths in my library path would not be an issue?

The issue happens as soon as any of these units are included in the uses clause:

//  test.orm.sqlite3         in '.\test.orm.sqlite3.pas';
//  test.orm.extdb           in '.\test.orm.extdb.pas';
//  test.orm.threads         in '.\test.orm.threads.pas';
//  test.orm.network         in '.\test.orm.network.pas';
//  test.soa.core            in '.\test.soa.core.pas';
//  test.soa.network         in '.\test.soa.network.pas';

So it will be something that is shared by all of them. I'll try to isolate it to a specific cause.

squirrel · mORMot 1

Hi ab

Not sure if this is relevant yet, but reporting incase it can be useful. Downloaded the latest release (172d480) from github's releases page and was able to successfully build and run mormot2tests for win32. No errors reported.

Changed mormot2tests to Win64 and when building using Delphi 10.3.1 on Win10 the compiler returns the following error:

[dcc64 Fatal Error] F2084 Internal Error: L3226

This is a Delphi internal error, so its probably an issue with Embarcadero's compiler. I dont see any other useful information to report. Restarting the ide did not remove the error.

Edit: forgot to mention that it also happens with the clone of the repository, not just the release

squirrel · mORMot 1

That works perfectly :-)

Thanks @ab. This is seriously impressive support.

squirrel · mORMot 1

It looks like another GotoNextNotSpace() might be needed in SynCommons.pas in the function TDynArray.LoadFromJSON function before line 50300, since that is where it gets the space value instead of the next [ character. Unfortunately I dont know the code well enough to know the impact of changes there.

Interesting is that in all scenarios, JSONArrayCount return the correct count.

mORMot Open Source

#1 Re: mORMot 2 » IDocList/IDocDict JSON for Delphi and FPC » 2024-02-06 06:55:34

#2 Re: mORMot 2 » Difference in URL parsing between mORMot1 and 2 » 2024-01-02 09:21:44

#3 Re: mORMot 2 » Difference in URL parsing between mORMot1 and 2 » 2023-12-08 06:33:03

#4 Re: mORMot 2 » Difference in URL parsing between mORMot1 and 2 » 2023-11-08 07:56:25

#5 Re: mORMot 2 » Difference in URL parsing between mORMot1 and 2 » 2023-11-08 06:27:56

#6 Re: mORMot 2 » Difference in URL parsing between mORMot1 and 2 » 2023-11-08 05:28:29

#7 Re: mORMot 2 » Difference in URL parsing between mORMot1 and 2 » 2023-11-07 11:11:00

#8 Re: mORMot 2 » Difference in URL parsing between mORMot1 and 2 » 2023-11-06 09:30:52

#9 Re: mORMot 2 » Difference in URL parsing between mORMot1 and 2 » 2023-10-27 13:36:16

#10 Re: mORMot 2 » Difference in URL parsing between mORMot1 and 2 » 2023-10-27 12:43:08

#11 mORMot 2 » Difference in URL parsing between mORMot1 and 2 » 2023-10-27 11:01:36

#12 Re: mORMot 2 » Blocking worker threads » 2023-09-29 10:06:39

#13 Re: mORMot 2 » Blocking worker threads » 2023-09-28 20:59:45

#14 mORMot 2 » Blocking worker threads » 2023-09-28 10:45:43

#15 mORMot 2 » CurrentServiceContext » 2023-09-18 11:58:52

#16 Re: mORMot 1 » Open source OAuth 2.0 Server implementation in Delphi by Carlos He » 2023-04-04 10:05:11

#17 Re: mORMot 1 » Implementing OAuth2 » 2023-04-04 08:41:43

#18 Re: mORMot 2 » Using LibMysql directly » 2023-02-23 08:46:58

#19 Re: mORMot 2 » Using LibMysql directly » 2023-02-21 10:51:06

#20 mORMot 2 » Using LibMysql directly » 2023-02-21 09:02:41

#21 mORMot 1 » Getting RowCount with ISQLDBRows » 2022-12-12 09:12:18

#22 Re: mORMot 1 » High-performance frameworks » 2022-08-22 08:47:11

#23 Re: mORMot 1 » Migrating to mormot 2 » 2022-08-15 07:33:56

#24 Re: mORMot 1 » Migrating to mormot 2 » 2022-08-11 15:47:13

#25 Re: mORMot 1 » Migrating to mormot 2 » 2022-08-11 15:25:04

#26 Re: mORMot 1 » Migrating to mormot 2 » 2022-08-11 12:07:30

#27 mORMot 1 » Migrating to mormot 2 » 2022-08-11 10:03:03

#28 Re: mORMot 1 » Implementing OAuth2 » 2022-07-19 12:38:25

#29 Re: mORMot 1 » Implementing OAuth2 » 2022-07-19 11:25:23

#30 Re: mORMot 1 » Stop the war! » 2022-07-19 09:00:19

#31 mORMot 1 » mORMot 2 Release » 2022-06-07 12:52:12

#32 Re: mORMot 1 » New Async HTTP/WebSocket Server on mORMot 2 » 2022-05-26 11:26:37

#33 Re: mORMot 1 » MVC URI routing » 2022-05-12 11:26:04

#34 Re: mORMot 1 » MVC URI routing » 2022-05-12 10:35:26

#35 Re: mORMot 1 » MVC URI routing » 2022-05-12 09:57:57

#36 Re: mORMot 1 » MVC URI routing » 2022-05-11 12:48:44

#37 Re: mORMot 1 » MVC URI routing » 2022-05-11 11:58:58

#38 Re: mORMot 1 » MVC URI routing » 2022-05-11 09:06:20

#39 Re: mORMot 1 » Setting content-type when returning blob » 2022-04-08 12:38:13

#40 Re: mORMot 1 » Setting content-type when returning blob » 2022-04-08 12:33:42

#41 mORMot 1 » Setting content-type when returning blob » 2022-04-08 09:49:19

#42 Re: mORMot 1 » Download big files from server » 2021-09-27 12:29:09

#43 Re: mORMot 1 » Mormot functions in a DLL » 2021-08-25 06:36:52

#44 Re: mORMot 1 » Mormot functions in a DLL » 2021-08-24 10:49:31

#45 Re: mORMot 1 » Mormot 2: Win64 Internal Error » 2021-05-18 13:28:23

#46 Re: mORMot 1 » Mormot 2: Win64 Internal Error » 2021-05-18 12:32:27

#47 Re: mORMot 1 » Mormot 2: Win64 Internal Error » 2021-05-18 11:31:19

#48 mORMot 1 » Mormot 2: Win64 Internal Error » 2021-05-18 11:14:21

#49 Re: mORMot 1 » Json Parsing issue » 2021-04-06 10:48:14

#50 Re: mORMot 1 » Json Parsing issue » 2021-04-06 10:30:16

Board footer