You are not logged in.
last commit Parse Real ip header in proxy still buggy,
fHttp.Headers is empty when ParseRemoteIPConnID called,
fix should do in THttpAsyncConnection.DoHeaders function ,not in THttpAsyncConnection.DoRequest,
this works in both rest and websocket , direct connection or proxy connection
function THttpAsyncConnection.DoHeaders: TPollAsyncSocketOnReadWrite;
var
status: integer;
//!!!! fix sample
procedure DoParseRealIP(var RemoteIP:RawUtf8);
begin
if fServer.fRemoteIPHeaderUpper <> '' then
FindNameValue(fHttp.Headers, pointer(fServer.fRemoteIPHeaderUpper),
RemoteIP, {keepnotfound=}true);
end;
begin
// finalize the headers
result := soClose;
if (nfHeadersParsed in fHttp.HeaderFlags) or
not fHttp.ParseCommand then
exit;
fHttp.ParseHeaderFinalize;
//!!!!! parse proxy x-forward-forx ..realip
//!!!!! just after fHttp.Headers retrieved before fServer.OnBeforeBody get called by following DecodeHeaders
//!!!!! we need to hnow everything in business layer callback like OnBeforeBody 、OnWebSocketUpgraded、OnWebSocketConnect、
//!!!!! OnWebSocketDisconnect etc... so we wont break things
DoParseRealIP(fRemoteIP);
// immediate reject of clearly invalid requests
status := DecodeHeaders; // may handle hfConnectionUpgrade when overriden
if status <> HTTP_SUCCESS then
begin
// on fatal error (e.g. OnBeforeBody) direct reject and close the connection
DoReject(status);
exit;
end;
and the fServer.fRemoteIPHeader should set to 'X-Forwarded-For:' , because FindNameValue does not process ':'
my fix is just a sample which ignore RemoteConnID parse
another problem, not sure is a bug :
unit mormot.net.async line 3323 ,when used as websocket server :
function THttpAsyncConnection.DoRequest: TPollAsyncSocketOnReadWrite;
var
output: PRawByteStringBuffer;
remoteID: THttpServerConnectionID;
sent: integer;
p: PByte;
flags: THttpServerRequestFlags;
begin
// check the status
if nfHeadersParsed in fHttp.HeaderFlags then
fServer.IncStat(grBodyReceived)
else
begin
// content-length was 0, so hrsGetBody* and DoHeaders() were not called
result := DoHeaders;
if (result <> soContinue) or
(fHttp.State = hrsUpgraded) then
exit; // rejected or upgraded <<<<<!!!
<<<<--- if our server is behind a reverse proxy . even we use the (fhttpServer).RemoteIPHeader :='X-Forwarded-For'
because we exit here ,following fServer.ParseRemoteIPConnID(fHttp.Headers, fRemoteIP, remoteid); never get a chance to exec
---->>
end;
// optionaly uncompress content
if fHttp.CompressContentEncoding >= 0 then
fHttp.UncompressData;
// prepare the HTTP/REST process reusing the THttpServerRequest instance
result := soClose;
remoteid := fHandle;
fServer.ParseRemoteIPConnID(fHttp.Headers, fRemoteIP, remoteid);
<<<<--- if our server is behind a reverse proxy . even we use the (fhttpServer).RemoteIPHeader :='X-Forwarded-For'
because we exit here ,following fServer.ParseRemoteIPConnID(fHttp.Headers, fRemoteIP, remoteid); never get a chance to exec
---->>
when we need to get the Real RemoteIP from header ,
those code works fine when it handle regular http rest request ,but not work in websocket mode.
in short
we should remove the 'Sec-WebSocket-Protocol:' section when return the 101 header if client dos not specify the SEC-WEBSOCKET-PROTOCOL in request header
Scenario 1:client specify the protocol in reqeust header
prot := Http.HeaderGetValue('SEC-WEBSOCKET-PROTOCOL');
P := pointer(prot);
if P <> nil then
these three line identify the protocol name from request Header,
then the following code should be:
Protocol := CloneByName(subprot, prot);
NOT
Protocol := CloneByName(subprot, uri);
i think it just a type error.
-------------------------------
Scenario 2:client do NOT specify the protocol in reqeust header,
then our code detect the protocol from url,
at the end our code return 101 header by this
FormatUtf8('HTTP/1.1 101 Switching Protocols'#13#10 +
'Upgrade: websocket'#13#10 +
'Connection: Upgrade'#13#10 +
'Sec-WebSocket-Connection-ID: %'#13#10 +
'Sec-WebSocket-Protocol: %'#13#10 +
'%Sec-WebSocket-Accept: %'#13#10#13#10,
[ConnectionID, Protocol.Name, extout,
BinToBase64Short(@Digest, SizeOf(Digest))], Response)
this all work fine if client make the request by put SEC-WEBSOCKET-PROTOCOL in header,
but if js code like this
ws = new WebSocket('ws://127.0.0.1/chat');
the brower will report error :
"Response must not include ‘Sec-WebSocket-Protocol’ header if not present in request;"
u can see it in chrome console.
there are many Scenarios we can not ask the client to specify the SEC-WEBSOCKET-PROTOCOL in header, like work with 3rd party.
unit mormot.net.ws.core
in TWebSocketProtocolList.ServerUpgrade proc
// identify the Websockets protocol
prot := Http.HeaderGetValue('SEC-WEBSOCKET-PROTOCOL');
P := pointer(prot);
if P <> nil then
begin
repeat
GetNextItemTrimed(P, ',', subprot);
Protocol := CloneByName(subprot, uri);
until (P = nil) or
(Protocol <> nil);
if (Protocol <> nil) and
(Protocol.Uri = '') and
not Protocol.ProcessHandshakeUri(prot) then
begin
Protocol.Free;
result := HTTP_NOTFOUND;
exit;
end;
end
line 2376 Protocol := CloneByName(subprot, uri); should be "Protocol := CloneByName(subprot, prot);"
and the final 101 upgrade response , if we use CloneByUri(uri) then
Response must NOT include 'Sec-WebSocket-Protocol' header if not present in request ,otherwise js client will fail(if it doesnt use the protocol header).
a blank project ,if it uses mormot.core.base, mormot.core.os units,
cross compile it with windows lazarus , it works on windows target .
but can NOT work on Linux 64 Target , error info :
/lib64/libc.so.6: version `GLIBC_2.34' not found (required by ./project)
/lib64/libc.so.6: version `GLIBC_2.34' not found (required by ./project)
if just uses the mormot.core.base, then it works fine.
there is topic about this https://forum.lazarus.freepascal.org/in … ic=58888.0
maybe the problem comes from mormot.core.os
laz ver :2.3.0 fpc ver:3.3.1 target os : x64 centos 7.9
test prj compile error on delphi 11 with latest commit from github
D:\Dev\rad\Lib\mORMot2\src\rest\mormot.rest.server.pas(3121,19): error E2003: E2003 Undeclared identifier: 'FromExternalQueryPerformanceCounters'
D:\Dev\rad\Lib\mORMot2\src\rest\mormot.rest.server.pas(3299,13): error E2017: E2017 Pointer type required
D:\Dev\rad\Lib\mORMot2\src\rest\mormot.rest.server.pas(3301,7): error E2003: E2003 Undeclared identifier: 'Changed'
D:\Dev\rad\Lib\mORMot2\src\rest\mormot.rest.server.pas(3302,13): error E2017: E2017 Pointer type required
D:\Dev\rad\Lib\mORMot2\src\rest\mormot.rest.server.pas(5944,9): error E2017: E2017 Pointer type required
D:\Dev\rad\Lib\mORMot2\src\rest\mormot.rest.server.pas(5949,5): error E2003: E2003 Undeclared identifier: 'Changed'
D:\Dev\rad\Lib\mORMot2\src\rest\mormot.rest.server.pas(5951,11): error E2017: E2017 Pointer type required
D:\Dev\rad\Lib\mORMot2\src\rest\mormot.rest.server.pas(5957,9): error E2017: E2017 Pointer type required
D:\Dev\rad\Lib\mORMot2\src\rest\mormot.rest.server.pas(5970,5): error E2003: E2003 Undeclared identifier: 'Changed'
D:\Dev\rad\Lib\mORMot2\src\rest\mormot.rest.server.pas(5972,11): error E2017: E2017 Pointer type required
D:\Dev\rad\Lib\mORMot2\src\rest\mormot.rest.server.pas(5986,9): error E2017: E2017 Pointer type required
D:\Dev\rad\Lib\mORMot2\src\rest\mormot.rest.server.pas(6000,11): error E2017: E2017 Pointer type required
D:\Dev\rad\Lib\mORMot2\src\rest\mormot.rest.server.pas(6010,11): error E2017: E2017 Pointer type required
D:\Dev\rad\Lib\mORMot2\src\rest\mormot.rest.server.pas(6015,9): error E2003: E2003 Undeclared identifier: 'Changed'
D:\Dev\rad\Lib\mORMot2\src\rest\mormot.rest.server.pas(6017,13): error E2017: E2017 Pointer type required
D:\Dev\rad\Lib\mORMot2\src\rest\mormot.rest.server.pas(6265,74): error E2250: E2250 There is no overloaded version of 'Enter' that can be called with these arguments
D:\Dev\rad\Lib\mORMot2\src\rest\mormot.rest.server.pas(6281,48): error E2008: E2008 Incompatible types
D:\Dev\rad\Lib\mORMot2\src\rest\mormot.rest.server.pas(6282,37): error E2015: E2015 Operator not applicable to this operand type
D:\Dev\rad\Lib\mORMot2\src\rest\mormot.rest.server.pas(7476,14): error E2003: E2003 Undeclared identifier: 'FromExternalQueryPerformanceCounters'
D:\Dev\rad\Lib\mORMot2\ex\mvc-blog\MVCModel.pas(735,2): error F2063: F2063 Could not compile used unit 'mormot.rest.server.pas'
some third party api need CAcertification, like cert.p12 ,cert.pem,cert_key.pem,
like some payout api,
is it a way to use CACert with TWinHttp Client?
and with this AS_UNIQUE , when use the ctxt.Returns(const NameValuePairs: array of const;....) to return some Torm data, those filed are all missing
AS_UNIQUE can mark a field value to be unique in the table,
but the woStoreStoredFalse option can not be set with
_ObjFast(const NameValuePairs: array of const) overload function,
therefor ,the filed marked as_unique won't be serialized ,it just missing.
i know i can use _ObjFast(aObject: TObject; aOptions: TTextWriterWriteObjectOptions = [woDontStoreDefault]) overload function,
but there are so mary scenarios _ObjFast(const NameValuePairs: array of const) comes more handy.
throuth the code ,woStoreStoredFalse is to protect some sensitive information like password,
the AS_UNIQUE should mean "unique", not "sensitive"
may be we need another marker to do so.
answer excepted , the problem js my orm record contain a nested TSQLRecord field, witch did not filled by the TSQLRecord.create
sorry about my expression , the "record" i mean a orm record object instance .
so the problem is _ObjFast with array of const params(with object ), the same code works whit 1.18 but not mormot2.
_ObjFast(['UserInfo',aUserRec,'Account',aAdminAccountrec]);
function _ObjFast(const NameValuePairs: array of const): variant; overload;
eg:_ObjFast(['UserInfo',aUserRecObj,'Account',aAdminAccountrecObj]);
same code works fine in momrot 1.18
env: delphi rad 10.4.2
ab wrote:Please check https://synopse.info/fossil/info/86e1418dc1
Works like a charm <3
ServiceManagerApp := TMypServiceManagerApplication.Create; ServiceManagerApp.Start(RestServer, TypeInfo(IMypServiceManagerApplication)); RestServer.ServiceMethodRegisterPublishedMethods('svc_', ServiceManagerApp);
and the App-Class has methods like list_session and may now be called as domain/root/svc/list/session :-)
your situation is too add one more level suburl to the mvc app above the TMVCRunOnRestServer.Create(Self,nil,'svc');
your case is to rewrite the list_session method to /list/session
still get me wrong.
i mean TMVCApplication alreay have this feture , but any regular class registerd by RestServer.ServiceMethodRegisterPublishedMethods('some') dose not .
eg:
type
TMyDemoClass = class
published
procedure list(Ctxt:TSQLRestServerURIContext);
end;
var
_demo:TMyDemoClass ;
begin
... rest root eg :root
RestServer.ServiceMethodRegisterPublishedMethods('demo', _demo);
end;
then i need to access the list method by http://domain/root/demo/list
but with current mormot implemention , it only can be accessed by http://domain/root/demolist
my point is to group all method with one Prefix.
this is not to say i need to register a method with / slash or _ underline
why TMVCApplication can register a prefix like "blog" and then can access it with "root/blog/default" ? i mean, why this feture can not be used in ServiceMethodRegisterPublishedMethods ? any class with its published methods registed with ServiceMethodRegisterPublishedMethods with prefix ,eg "test" ,why we can not access it use "root/test/*" but "root/test*"?
i'd like to use
xx:=formatUtf8('abc%',[i])
but if the mormot.core.json and mormot.core.text both imported in the unit , u can't do so. it can NOT be compiled .
i had to remove the mormot.core.json use from the unit.
when both core.json and core.text in the uses cluase, compiler can not find the right FormatUtf8 func
function FormatUtf8(const Format: RawUtf8; const Args: array of const): RawUtf8; overload;
if it does ,the the uri will be more friendly
eg: domain/root
domain/root/admin/*
domain/root/api/*
i know the interface based service can do this ,but some times ,the method based service can be more flexable
is there a way to use the orm without restserver or restclient or any rest*?
just wanna to use the orm direct
seems very good
mOMRot2 test fronzen dead
after showing this :Ini files: 7,028 assertions passed
if remove "TTestCoreProcess" from then it fronzen after "Url decoding: 1,101 assertions passed"
env: delphi 10.4.1 windows 10 ltsc win32 debug mode
First call --> Server A
Second call --> Server B (requires root/auth login to get new session signature)
Third call --> Server A (session signature was changed above, so call is now rejected from server A)
---------------------------
this is why Redis become popular. a distributed session.
thanks for the quick reply .
so , if i DONT need to reuse the json, there is no need to call UniqueRawUtf8,
just
vObj :=TMyObject(JSONToNewObject(Pointor(vJson),vObjValid,[j2oIgnoreUnknownProperty]));
will be ok ? am i right?
i test JSONToObject function ,it works , but seems to change the json content.
i want to reuse the json content ,
follow the Documentation's guide ,
i make a local copy use @vJson[1] or UniqueRawUTF8
but after JSONToObject ,i write the vJson variable content to a memo , it only contains "classname"
var
vJson:rawutf8;
vJsonP:PUTF8Char;
begin
vJsonP :=UniqueRawUTF8(vJson);//@vJson[1];
vObj :=TMyObject(JSONToNewObject(vJsonP,vObjValid,[j2oIgnoreUnknownProperty]));
memo1.text:=vJson; //<--here it only output "classname" not the whole json content
end;
seems local copy did not work
may be ,we need a MultiReadExclusiveWrite / MultiReadSingleWrite kind of lock in mormot code base.
like Delphi 10.4 TLightweightMREW wrapper or TSpinLock.
or like this repo:https://github.com/BeRo1985/pasmp
Database created by mORMot Framework can Not be operated by 3rd tools
like Navicat
take Sqlite3\sample\ 30 - MVC Server for demo,
navicat can open and do select Query operation
but write operation like Insert will trigger an error:
no such collation sequence: SYSTEMNOCASE
line 26551 to line 26561 can not compile
prod := SysUtils.Trim(ReadString('SystemBiosVersion'));
env:windows 10 (not virtual machine) delphi 10.3.2
I'd like to implement ByPassAuthentication when use JWT authentication ,
so i create my own ServicesRouting class and override the AuthenticationCheck(jwt: TJWTAbstract) procedure,
but the fPublishedMethod and fIPWhiteJWT is not accessable ,
may i suggest add two property'PublishedMethod' and 'IPWhiteJWT' to the TSQLRestServer class?
Well, got it now, we should disable the Authentication on RestServer Creation first..
Just Asigned a TJWTHS256 instance to the RestServer.JWTForUnauthenticatedRequest property,
then set the client SessionHttpHeader :Authorization: Bearer <Token>
then call any method or interface based service that need auth ,you will fail.
problem is that :
JWTForUnauthenticatedRequest not working since procedure TSQLRestServer.URI security handling logic bug:
in mOMRot.pas:
// 2. handle security
if (rsoSecureConnectionRequired in fOptions) and
(Ctxt.MethodIndex<>fPublishedMethodTimestampIndex) and
not (llfSecured in Call.LowLevelFlags) then
Ctxt.AuthenticationFailed(afSecureConnectionRequired) else
if not Ctxt.Authenticate then
Ctxt.AuthenticationFailed(afInvalidSignature) else
if (Ctxt.Service<>nil) and
not (reService in Call.RestAccessRights^.AllowRemoteExecute) then
if (rsoRedirectForbiddenToAuth in Options) and (Ctxt.ClientKind=ckAjax) then
Ctxt.Redirect(Model.Root+'/auth') else
Ctxt.AuthenticationFailed(afRemoteServiceExecutionNotAllowed) else
if (Ctxt.Session<>CONST_AUTHENTICATION_NOT_USED) or
(fJWTForUnauthenticatedRequest=nil) or
(Ctxt.MethodIndex=fPublishedMethodTimestampIndex) or
((llfSecured in Call.LowLevelFlags) and
not (llfHttps in Call.LowLevelFlags)) or // HTTPS does not authenticate
Ctxt.AuthenticationCheck(fJWTForUnauthenticatedRequest) then
line 42740 42741 always get executed if not authed
if not Ctxt.Authenticate then
Ctxt.AuthenticationFailed(afInvalidSignature)
so fJWTForUnauthenticatedRequest never get a chance .
or i am doing it wrong?
keinn wrote:i know,i know , u can make AJAX request to the RestServer (not to the mvc application),
but this approach will need to deal with the rest auth, while MVC application is based on Cookie.
You don't need to deal with rest auth, you can disable/ignore it and deal with cookies directly.
you mean process Cookie head(Read/Write) in the Ctxt Context within the method based service or in the ServiceContext within the interface based service?
this can do it , and i know it , reason i post this thread is hope mORMot MVC will be more powerful and do not rely on other parts of the framework:
lets say :
mORMotMVC is more suitable to deal with cookie\Session ,while Rest Service is not so direct . and also , it is not in the MVC subRoot
again within the mORMotMVC , if we can use the Service ability directly(do not need view) in the TMVCApplication , it will be perfect
for now , mORMot MVC have some problem :
1、the methodname/json/ can return json data , but it can NOT take param(like json format) send by POST ,just GET work for now
2、every TMVCApplication view method need a view template, while some processing like :login/logout just need process params(by get/post) , they do NOT need view template
Thanks ab for the great mORMot framework, it helps me a lot.
i use the framework SOA、ORM a lot .
now i need to build a web site. i guess at somepoint , we all need a website.
i found a little imperfect of MVC:
when you just need to show some pages,
just write you view template and define some interface , it works fine, same as Sample 30--MVC Server.
but, when you need to use Ajax in pages to Post data to or Get data from the mvc server, it CAN NOT be done.
i mean , when you need to do AJAX request from the MVC application, there the NO WAY to return json data from the mvc application(interface)
i know,i know , u can make AJAX request to the RestServer (not to the mvc application),
but this approach will need to deal with the rest auth, while MVC application is based on Cookie.
for example:
when user need to login to the Web(not the RestSerrver), since our MVC application can not return custom json format (which ajax need),
we loose the ablity to interact with user before or after the login.
or , some page need use Ajax request data only after user login (again, web cookie, not the restserver auth)
so, if mvc application return JSON data will solve all that .(not the methodname/json approach, it is still for view context)
similar question here:
when doing such Batch Insert (already checked duplicates), or Single Insert many times (eg: insert 1000 rows of data ),
we need to hnow if the data already exsist in DB,
if we use
TSQLRecord*.Create(aRest,'id=?',[i])
to check first ,the insert process will be very very slow, the already exist data may or maynot need update...
line 51791 of SynCommons.pas:this line:
if not RegisterCustomJSONSerializerSetOptions(aTypeInfo[XXXXX],aOptions) then
Missing the "aAddIfNotExisting" parameter cause the register of array of aTypeInfo failing.
also:i use aTypeInfo[XXXXX] ,xxxx replace i in code above because when use i in this post ,i can NOT post.
i know that now, the key also should uses some digest method to generate
var
ret:rawbytestring;
display:rawutf8;
begin
aes:=TAESECB.Create('123',128);
try
ret :=aes.EncryptPKCS7('mORMot is awesome!');
display :=BinToHex(ret);
finally
aes.free;
end;
end;
here the display result is 76D7EC0CD8E8EACA15F8214126A62D3A7C503EB0820B52233986FF917554669C
it is wrong.
when use 128 KeySize , the result should be 1d35c8513d106079e66c1f13a0e57eb5d96f97231f345b07bd42ae6d9dd02694
input string data:"mORMot is awesome!", password is "123"
the result AESECB with pkcs5padding hex result should be :
"1d35c8513d106079e66c1f13a0e57eb5d96f97231f345b07bd42ae6d9dd02694"
is there some Decrypt wrapper function like :
function DecryptAesEcbFromHexString(const HexStr,Password:RawUtf8):RawUTF8;
or Encrpyt string to Hex string like:
function AesEcbEncryptAsHex(const Str,Password:RawUtf8):RawUTF8;
can u show some pieces of code please?
thank you very much .
Yes~~i just find about SynZip.pas ~
thank you for the lovely and also fastest reply ~
for example ,
hex:=SynCommons.crc32cUTF8ToHex('1');
here the result is '90F599E3'
it is diff from the widely used online service
https://www.lammertbies.nl/comm/info/cr … ation.html
and other online calculation services
it should be '83DCEFB7'
I use serval online enc/dec services to check the AESECB encryption/decryption result ,
it seems that our SynCrypto TAESECB mode can not get any result same with the online services.
what is your result when do such enc/dec with TAESECB?
the utf8 string key is '123456'
utf8String Data is 'ABC'
All online services results:
PKCS7padding, keysize 256 : h3ivt+HEEOkunT75PoB199xf3tD1LRO/p1/l6Tdrg6I=
PKCS7padding, keysize 128 : FevVdP5MovJAiVEClsvBBQ==
none of them is same as our syncrypto results.
my code like this :
CONST
KEY:RawByteString='123456';
VAR
aKey: TSHA256Digest;
AES:TAESECB;
aa:TAESCFB;
DAT:RawByteString;
sr:RawByteString;
begin
DAT:='ABC';
aKey :=SHA256Digest(KEY);
AES:=TAESECB.Create(aKey,256);
sr:=AES.EncryptPKCS7(DAT);
mmoLog.Lines.Add(BinToBase64(sr));
end;
result is EUouYChaLyxO7ArP0k3G4A==
what is your result with syncrypto?
We created some SOA rest servers use TSQLRestServerDB,
Like :OrderRestServer,MemberRestServer,ProductRestServer,DeliverRestServer ,etc..
Each RestServer Running in different process deployed at different machine.
So we need to create one TSQLHttpClient for each *RestServer, so that is more then 5 TSQLHttpClient instance at client side (mobile app for example)
If app client need to get some data like: get all orders and delivery status of one member.
That's just so many calls need to made by app.
We think that is the wrong approach.
Is there a easy way to implement a Rest Api GateWay (like SpringCloud-Zuul or Kong), which combine many backend Rest Api within one http call?
Or is it the right way to do it like this: Use a dedicated Front Rest Server to provide the Aggregated Api , in this aggreated api ,we manually call all needed backend rest apis, then return the combined data to client?
(but this approach need the Front Aggregate Rest Server to create so many httpclient instance to the backend rest server for each single client request)
Is there a better(efficient) way?(mORMot way of course)
Happy new year by the way.
i pull the latest source from git, still cannot compile anything with 10.3(Rio).
ide stopped at line 18794 in SynCommon.pas
error message:
[dcc32 Error] SynCommons.pas(18794): E2015 Operator not applicable to this operand type
ide stopped at
procedure TSynTempBuffer.Init(Source: pointer; SourceLen: integer);
begin
len := SourceLen;
if len<=0 then
buf := nil else begin
if len<=SizeOf(tmp)-16 then
buf := @tmp else
GetMem(buf,len+16); // +16 for trailing #0 and for PInteger() parsing
MoveFast(Source^,buf^,len);
PPtrInt(buf+len)^ := 0; // always init last 4/8 bytes (makes valgrid happy) <<----------stopped here
end;
end;
all 36 demo projects failed
i pull the latest source from git, still cannot compile anything with 10.3(Rio).
same problem , ide stopped at line 18794 in SynCommon.pas : PPtrInt(buf+len)^ := 0; // always init last 4/8 bytes (makes valgrid happy)
we have similar usage like this:
https://synopse.info/forum/viewtopic.php?id=4712
many TSQLRestServerDB instance hosted in one httpserver,
we do this to preform "Each Client Has It's Own Database", and to practice the "Sharding"
this approach really simplify the project's architecture and coding.
we have 1000 TSQLRestServerDB s hold in each Httpserver ,yes you see it right , 1000+
by test we fill two table of each database with One Million rows ,then each db file in disk is about 200M
we load all the 1000 databases, the project's whole RAM use is just around 300M, so we think with todays machine ablility ,
this will work just fine .
and we run 10+ process on each machine to get maxium use of the machine power.
if this work ok ,it will save a lot of money!!
after reading the linked post above , we afriad this approach may cause some other issue, like backup\ cache\ master\slave etc
any thought?
hi,everyone
i need to update the table structure, add some more fileds to it,
but if we just add the new filed definition to the TSQLRecord,
the default value will be NULL at all old table row( old table have many rows ,not empty new table),
in practice , we need to fill default value to it .
i try the InitializeTable proc, but cannot find a proper way to do it .
anyone can post a sample code?
thanks very much.
for example, our online services has many Couppon infomation , currently we use the ORM Rest Part of mORMot to serve the requests from all kinds of clients, such as browser ajax, mobile apps , pc clients.
something like this:
TSQLCoupponInfo = class(TSQLRecord)
private
fId: TID;
fProductName: RawUTF8;
fManufactor: RawUTF8;
fPrice:Currency;
fCouppon:Currency;
fDueTime:TDateTime;
fCreatedAt: TCreateTime;
fModifiedAt: TModTime;
etc...
published
property id:TID read fId;
property ProductName :rawuft8 read fProductName write ProductName;
etc ...
property CreatedAt: TCreateTime read fCreatedAt write fCreatedAt;
property ModifiedAt: TModTime read fModifiedAt write fModifiedAt;
end;
many table like this contains more then 20 fileds,and such table row count growing so fast (nearly 200 thousand per day),
now it contains nearly 5million rows .
the valid Couppon info is just the recent few days, we do not need to lookup the whole table,
but mORMot does not have a way to do something like :generate one table per day (or other strategy)
by this project, we build it all the mORMot rest way, no stored proc, few table relation etc.
now we facing the rapid growing table size and massive client request(nearly 1million anonymous UV, 10million PV) , and our marketing invest will make the load way more heavy in the foreseeable future. not just the public services, many other services need auth also faceing the heavy requests.
now only thing we do ,it's just put many CDN server at Services's frontend and use more powerful machine.
we go through all the mORMot doc and forum info, we just found AB said that :you dont need scale or loadbalancing.
but we are so afraid that one day , our services just can't hold the request load anymore.
so, we need such sugestion:
1. table scale to control single table row count;
2. Rest Services load balancing.
and another thing , rest services really need version control .
we only use delphi to build or whole project, only 3 coder , and we love delphi/pascal , we really dont wanna switch to Java or Go solution.
maybe we used mORMot the wrong way . but any advice will be appreciated.
thanks very much.
i mean, i suffered "client not response" issue, when using the TSQLHttpClientWebsockets。
when the client callback deal with long time process, it will block the websocket connection.
detailed here: https://synopse.info/forum/viewtopic.php?id=4547
is there a way to post the callback process to a dedicated backgroud thread? other than Client.ServiceNotificationMethodViaMessages()?,
sometimes there is no window form to process Tmessage loop.
after deep dive into the synopse online doc, i think , the Client.ServiceNotificationMethodViaMessages(); will solve the problem.
but , another case, i use the websocket client in dll , and the dll is loaded by a third party exe ,not coded in delphi~~ wanna cry.
question:how the webscoket client callback work? is it running in thread ? how to perform NoBlocking on client side?
while testing the sample 31: Project31ChatServer.dpr and Project31ChatClient.dpr. the default code works .
but if made one small change on the client side callback as such:
procedure TChatCallback.NotifyBlaBla(const pseudo, msg: string);
begin
TextColor(ccLightBlue);
writeln(#13'@',pseudo,' ',msg);
TextColor(ccLightGray);
write('>');
sleep(60000); //just test the slow process
end;
and modify the procedure Run; make it send 10000 Blabla Requests as such:
TextColor(ccLightGray);
write('>');
readln(msg);
if msg='' then exit;
i:=0;
repeat
Service.BlaBla(pseudo,msg);
until i=10000;
run such modified client , and the UnModified client together, just for a short while, the client will raize:
TInterfacedObjectFakeClient.FakeCall(IChatService.BlaBla) failed: 'URI root/ChatService.BlaBla ["888888888888","88888888888"] returned status 'Not Found' (404 - Network problem or request timeout)'
it seems due to the TChatCallback.NotifyBlaBla callback service take too long to process ,the webscoket client is closed by the server .
so my question is : when it's call by server, what thread does the websocket callback implemetion running in? ie:TChatCallback.NotifyBlaBla(const pseudo, msg: string);
is it thread safe? and seems "NoBlocking" is not implement on both side