You are not logged in.
Alex, if you send me your email address, i'll mail the units for CRC32 and SHA256 to you
I have managed to create a VBA method that authenticates to a REST server, it stores the resulting PrivateHash, SessionID etc in global vars:
Option Explicit
Dim gSessionIDasHex As String
Dim gPrivateSaltHash As Long
Dim gPreviousTix As Long
'Format number as a 4 Byte hex
Private Function To4ByteHex(aNr As Long) As String
Dim AsHex As String
AsHex = Hex(aNr)
To4ByteHex = Right("00000000" + AsHex, 8)
End Function
Private Function Authenticate(ByVal aServer, aModel As String, ByVal aUsername, aPassword As String) As Boolean
Dim MyRequest As WinHttpRequest
Dim Response As String
Dim ServerNonce As String
Dim PasswordHashHexa As String
Dim SessionPassWordSha256 As String
Dim ClientNonce As String
Dim s, i As Integer
Set MyRequest = New WinHttpRequest
gSessionID = 0
gSessionIDasHex = ""
gPrivateSaltHash = 0
'First request ServerNonce for username
MyRequest.Open "GET", aServer + "/" + aModel + "/Auth?username=" + aUsername
MyRequest.Send
If MyRequest.Status <> 200 Then
Debug.Print MyRequest.ResponseText
Authenticate = False
Else
Response = MyRequest.ResponseText
'Authenticate and create session
ServerNonce = Mid(Response, InStr(Response, ":") + 2, 64)
ClientNonce = SHA256("TestingCLient")
PasswordHashHexa = SHA256("salt" + aPassword)
SessionPassWordSha256 = SHA256(aModel + ServerNonce + ClientNonce + aUsername + PasswordHashHexa)
MyRequest.Open "GET", aServer + "/" + aModel + "/auth?userName=" + aUsername + "&PassWord=" + SessionPassWordSha256 + "&ClientNonce=" + ClientNonce
MyRequest.Send
If MyRequest.Status <> 200 Then
Debug.Print MyRequest.ResponseText
Authenticate = False
Else
s = InStr(MyRequest.ResponseText, ":")
i = InStr(MyRequest.ResponseText, ",")
'gPrivateSalt is "<SessionID>+<PrivateKey>"
gPrivateSalt = Mid(MyRequest.ResponseText, s + 2, i - s - 3)
i = InStr(gPrivateSalt, "+")
gSessionID = Val(Left(gPrivateSalt, i - 1))
gSessionIDasHex = To4ByteHex(gSessionID)
gPrivateSaltHash = AddCrc32(PasswordHashHexa, AddCrc32(gPrivateSalt))
Authenticate = True
End If
End If
End Function
After authenticating, yu can use the folliwng sub to append the needed signature to the url
'Construct the signature to add to the url,
Public Function SignUrl(Url As String) As String
SignUrl = SignUrlEx(Url, gSessionIDasHex, gPrivateSaltHash)
End Function
Public Function SignUrlEx(ByVal Url As String, ByVal SessionIDAsHex As String, ByVal PrivateSaltHash As Long) As String
Dim Sig As String
Dim Tix As Long
Dim Timestamp As String
Sig = "session_signature=" + SessionIDAsHex
Tix = Timer * 100
'//Must be larger then prvious request
If Tix <= gPreviousTix Then
Tix = gPreviousTix + 1
End If
gPreviousTix = Tix
Timestamp = To4ByteHex(Tix)
Sig = Sig + Timestamp
Sig = Sig + To4ByteHex(GetCrc32(AddCrc32(Url, AddCrc32(Timestamp, PrivateSaltHash))))
If InStr(Url, "?") > 0 Then
SignUrlEx = Url + "&" + Sig
Else
SignUrlEx = Url + "?" + Sig
End If
End Function
So you can test this with the following sub:
Public Sub TestUrl()
Dim MyRequest As WinHttpRequest
Dim Response As String
Dim Server As String
Dim Model As String
Dim Url As String
Set MyRequest = New WinHttpRequest
Server = "http://localhost:8080"
Model = "root"
If Authenticate(Server, Model, "User", "synopse") Then
Url = Model + "/SampleRecord"
MyRequest.Open "Get", Server + "/" + SignUrl(Url)
MyRequest.Send
Debug.Print MyRequest.ResponseText
Url = Model + "/SampleRecord/2"
MyRequest.Open "Get", Server + "/" + SignUrl(Url)
MyRequest.Send
Debug.Print MyRequest.ResponseText
End If
End Sub
If you need them I can give you the CRC32 and Sha256 methods too, just let me know.
Ohh .. never mind ...
I was reading the (always updated) manual, but using outdated software
Reading the SAD on page 300 (paragraph 17.7.4.1.1) it states:"
As an alternative, you can encode and name the parameters at URI level, in a regular HTML fashion:
GET root/Calculator.Add?n1=1&n2=2
Since parameters are named, they can be in any order. And if any parameter is missing, it will be
replaced by its default value (e.g. 0 for a number or '' for a string).
This doesn't seem to work, tracing the call from the RestServer gives the following callstack
Mormot_D19.mORMot.TServiceMethod.InternalExecute((...),'Username=jan&aPassword=piet&session_signature=1073BBD7004149EDA3E6D51A',$FE43DB20,'',[],True)
Mormot_D19.mORMot.TServiceFactoryServer.ExecuteMethod($FE7EE260)
Mormot_D19.mORMot.TSQLRestServerURIContext.InternalExecuteSOAByInterface
Mormot_D19.mORMot.TSQLRestRoutingREST.ExecuteSOAByInterface
Mormot_D19.mORMot.TSQLRestServer.URI($76EFDC4)
Mormot_D19.mORMotHttpServer.TSQLHttpServer.Request($FE380100)
Mormot_D19.SynCrtSock.THttpServerGeneric.Request($FE380100)
Mormot_D19.SynCrtSock.THttpApiServer.Execute
Looking at line 34544 the TServiceMethod.InternalExecute() clearly only accepts parameters starting with a '['
We are investigating the possibilities of accessing data in our server application from within Excel, especially using PowerQuery.
We did manage to use normal authentication scheme from Mormot by writing VBA functions to handle the comminication and the calculation the session_signature parameter. Sadly, there is no way from PowerQuery to call VBA functions. Probably has to do with the possibility to store theses sheets on the cloud, where powerQueries can still be executed, but VBA methods cannot.
Main problem is the fact that (with the same REST command) the actual URI will be different every time because it must contain the (ever increasing) timestamp.
A widely spread alternative way to access data via REST is to use an ApiKey as a kind of special password. We want to implement this using a special call to ask the server for a ApiKey for a specific user (stored in the TSQLAuthUser table), which will timeout when not used for some time. As long as the timeout has not expired, the ApiKey can be used as a parameter in the URI and provide authentication.
http://localhost:8080/API/GetApiKey?User=<username>&password=<sha256encodedpassword>
This call will return a valid ApiKey (when the credentials check out of course). With this Apikey all other available REST command should now be available (without any additional registration of published methods or interfaces) by calling them like this:
http://localhost:8080/API/SampleRecord?ApiKey=<ApiKey>
http://localhost:8080/API/SampleRecord/1?ApiKey=<ApiKey>
http://localhost:8080/API/Sum?ApiKey=<ApiKey>&a=23&b=12
What would be the best/easiest/most robust way to implement this?
Note: To provide extra security use of https would be advised of course, but that is irrelevant to the concept
Guess you misunderstood my question. We are trying to get the data from a Mormot TSQLRestServer into Excel 2013, and then use Excel PowerQuery and powerView to analyze the data report on it
We are investigating the possibilities to use Excel 2013 Power Query (and Pivot, Map etc) as the reporting and dashboard tool for our applications data.
Main aspect of course is how to retrieve the data made available by our server application via a TSQLRest service and store it in the Excel data model, maybe even share it via Office365. Another challenge would be to find a way to write changing parameters from the Excel sheet PowerQueries into our
Using TestProject04 we have been able to retrieve data from the HTTP server using power queries "From Web" function and normal URLs like http://localhost:8080/root/SampleRecord but this seem to only work when authentication is disabled. Can this method be used when authentication is enabled?
Using ODATA would be another possibility, but reading the posts on this subject on this forum I understand that AB is not a great supporter of that "standard"
Anybody any tips on how to accomplish this?
Thanks
Bas
Added the ticket
I'v defined a TSynLog in our server application, and would like to be able to retrieve the logged records from our client application using ORM and the TSQLRecordLogFile definition.
How can this be achieved?
After downloading your last commit, implementing ISQLDBStatement i get compile errors in MormotDB:
[dcc32 Error] mORMotDB.pas(881): E2010 Incompatible types: 'TSQLDBStatement' and 'ISQLDBStatement'
[dcc32 Error] mORMotDB.pas(1113): E2010 Incompatible types: 'TSQLDBStatement' and 'ISQLDBStatement'
[dcc32 Error] mORMotDB.pas(1169): E2010 Incompatible types: 'TSQLDBStatement' and 'ISQLDBStatement'
[dcc32 Error] mORMotDB.pas(1211): E2010 Incompatible types: 'TSQLDBStatement' and 'ISQLDBStatement'
[dcc32 Error] mORMotDB.pas(1231): E2010 Incompatible types: 'TSQLDBStatement' and 'ISQLDBStatement'
Cause by variable Query declared as TSQLDBStatement in stead of ISQLDBstatement
Ticket: http://synopse.info/fossil/tktview/4332 … f207c1a1e6
Edit:
After changing all TSQLDBstatement declarations to ISQLDBstatment, I'm also getting lots of access violations in the selftests, part 2.7 External Database .. maybe you have forgotten to commit one or two changes?
Stacktrace:
SynSQLite3.TSQLRequest.Step
SynSQLite3.TSQLRequest.Execute(0,'',$213BE78,True)
mORMotSQLite3.TSQLRestServerDB.EngineList('SELECT RowID,FirstName,LastName,YearOfBirth,YearOfDeath,LastChange,CreatedAt FROM PeopleExt WHERE RowID=:(1):;',True,nil)
mORMotSQLite3.TSQLRestServerDB.EngineRetrieve(1,1)
mORMot.TSQLRestServer.URI($18FB80)
mORMotSQLite3.TSQLRestClientDB.InternalURI('root/PeopleExt/1','GET',$18FCB8 {''},nil {''},nil {''})
mORMot.TSQLRestClientURI.URI('root/PeopleExt/1','GET',$18FCB8 {''},nil {''},nil {''})
mORMot.TSQLRestClientURI.URIGet(TSQLRecordPeopleExt,1,'',False)
mORMot.TSQLRestClientURI.EngineRetrieve(1,1,False,6,'')
mORMot.TSQLRestClient.Retrieve(1,$21691B0,False)
SynSelfTests.TTestExternalDatabase.Test(True)
SynSelfTests.TTestExternalDatabase.ExternalViaREST
SynCommons.TSynTests.Run
mORMotSelfTests.SQLite3ConsoleTests
TestSQL3.TestSQL3
Again .. we managed to create our own problems here hahaha
The above problem was caused by the TSQLDBNexusDBStatement.ColumnTypeNativeToDB() implementation. There the DB types ftMemo which is used for unindexed UTF8 fields was mapped to SynDB.ftBlob in stead to SynDB.ftUTF8 ...
Does raise the question why these Object serializations into TEXT fields are always read by default (unlike regular Blob fields).. a serialized collection can get quite large ..
I have created a TSQLRecord with a TInterfaceCollection as one of its published properties.
When this property is written to the underlying database (our own NexusDB implementation in this case) it results in a neat JSON string
[{"ValueDataObject":"X'6F626A65637420544C435353696D706C6556616C75650D0A20204173537472696E67203D20274469742069732065656E2074656B73746A65270D0A656E640D0A'","AttrDefDataObjct":"X'6F626A6563742054574244796E4174747269627574654465660D0A20204E616D65203D202756616C75654E616D6531270D0A20205742417474726962757465436C617373203D2027544C435353696D706C6556616C7565270D0A656E640D0A'"},{"ValueDataObject":"X'6F626A65637420544C435353696D706C6556616C75650D0A20204173466C6F6174203D2033342E3636353030303030303030303030303030300D0A656E640D0A'","AttrDefDataObjct":"X'6F626A6563742054574244796E4174747269627574654465660D0A20204E616D65203D202756616C75654E616D6532270D0A20205742417474726962757465436C617373203D2027544C435353696D706C6556616C7565270D0A656E640D0A'"}]
When i try to retrieve a record from this model, it does not work anymore :-(
The JSONToObject function in Mormot.pas is called from TSQLPropInfoRTTIObject.SetValue with the following value of From:
'ï¿°W3siVmFsdWVEYXRhT2JqZWN0IjoiWCc2RjYyNkE2NTYzNzQyMDU0NEM0MzUzNTM2OTZENzA2QzY1NTY2MTZDNzU2NTBEMEEyMDIwNDE3MzUzNzQ3MjY5NkU2NzIwM0QyMDI3NDQ2OTc0MjA2OTczMjA2NTY1NkUyMDc0NjU2QjczNzQ2QTY1MjcwRDBBNjU2RTY0MEQwQSciLCJBdHRyRGVmRGF0YU9iamN0IjoiWCc2RjYyNkE2NTYzNzQyMDU0NTc0MjQ0Nzk2RTQxNzQ3NDcyNjk2Mjc1NzQ2NTQ0NjU2NjBEMEEyMDIwNEU2MTZENjUyMDNEMjAyNzU2NjE2Qzc1NjU0RTYxNkQ2NTMxMjcwRDBBMjAyMDU3NDI0MTc0NzQ3MjY5NjI3NTc0NjU0MzZDNjE3MzczMjAzRDIwMjc1NDRDNDM1MzUzNjk2RDcwNkM2NTU2NjE2Qzc1NjUyNzBEMEE2NTZFNjQwRDBBJyJ9LHsiVmFsdWVEYXRhT2JqZWN0IjoiWCc2RjYyNkE2NTYzNzQyMDU0NEM0MzUzNTM2OTZENzA2QzY1NTY2MTZDNzU2NTBEMEEyMDIwNDE3MzQ2NkM2RjYxNzQyMDNEMjAzMzM0MkUzNjM2MzUzMDMwMzAzMDMwMzAzMDMwMzAzMDMwMzAzMDMwMzAwRDBBNjU2RTY0MEQwQSciLCJBdHRyRGVmRGF0YU9iamN0IjoiWCc2RjYyNkE2NTYzNzQyMDU0NTc0MjQ0Nzk2RTQxNzQ3NDcyNjk2Mjc1NzQ2NTQ0NjU2NjBEMEEyMDIwNEU2MTZENjUyMDNEMjAyNzU2NjE2Qzc1NjU0RTYxNkQ2NTMyMjcwRDBBMjAyMDU3NDI0MTc0NzQ3MjY5NjI3NTc0NjU0MzZDNjE3MzczMjAzRDIwMjc1NDRDNDM1MzUzNjk2RDcwNkM2NTU2NjE2Qzc1NjUyNzBEMEE2NTZFNjQwRDBBJyJ9XQ=='
As far as i can see this is the result after calling WRBase64() in the Writer, and this is not reversed when reading.
There was another strange thing I noticed while debugging this:
When TSQLRecordProperties.Create() is called, fields with an SQLFieldType of sftObject are handled as a SimpleField, but they are actually stored as a TEXT field (Blob Memo in the case of our NexusDB implementation). Doesnt this create problems when reading back, as I had expected these properties of unknown size to be read only after calling RetrieveBlobFields()... ?
Greetings
Bas
Mmm been complaining here too soon :-( ...
was caused by our own implementation for the NexusDB
sorry!
If I define a published TCollection property in a TSQLRecord, the corresponding field will be created as a Widestring(4000). This limits the number of items the collection can contain.
Is there a special reason why a collection is not stored as a TEXT or BLOB field?
Already created a ticket: http://synopse.info/fossil/info/f99c86b1ac
Mmm .. too bad :-(
That will force us to go back to the Delphi streaming system
Further investigation learns that reading of the collection end in error when the TValColItem.AttributeDef value is read from the JSON stream.
To be more precise, the following call in mORMot.pas line 25742 results in nil
PObject(P^.GetFieldAddr(Value))^
The definition of TWBDynAttributeDef:
TWBDynAttributeDef = class(TLCSDataObject, IWBDynAttributeDef) {TLCSDataObject is derived from TPersistent}
private
FDisplayName: string;
FName: string;
FWBAttributeClass: string;
protected
function GetAttribClass: string;
function GetName: string;
procedure SetAttribClass(const aValue: string);
procedure SetName(const aValue: string);
published
property DisplayName: string read FDisplayName write FDisplayName;
property Name: string read GetName write SetName;
property WBAttributeClass: string read GetAttribClass write SetAttribClass;
end;
A question I have is, what happens when you use some TPersistent in TSQLRecord, and you assign and object for an inherited class to it.
TMyPersistent = class(TPersistent)
private
FName: String;
published
Name: string read Fname write Fname;
end;
TMyDerived = class(TMyPersistent)
private
FAddress: string;
published
Address: String read FAddress write FAddress;
end;
TMyRecord = class(TSQLRecord)
private
FPers: TMyPersistent;
published
Pers: TMyPersistent read FPers write FPers;
end;
What happens when i assign a instance of TMyDerived to TMyRecord.Pers and write that to a database? Will the Address property be written to?
What happens when i read TMyRecord back from the database? Will it create an TMyPersistent object or a TMyDerived object?
I have created a TSQLRecord class which contains a TCollection as one of its properties.In my test the Collection contains two items.
TSQLWorkbaseObject = class(TSQLWorkbaseBaseObject)
type
TValueColItem = class(TCollectionItem)
private
Fname: RawUTF8;
FAttrDef: TWBDynAttributeDef;
FValue: TLCSDataObject;
published
property Name: RawUTF8 index 50 read FName write FName;
property AttributeDef: TWBDynAttributeDef read FAttrDef write FAttrDef;
property ValueObject: TLCSDataObject read FValue write FValue;
end;
TValueCollection = class(TCollection)
private
function GetCollItem(Index: Integer): TValueColItem;
public
function Add: TValueColItem;
property ItemIndex: Integer: TValueColItem read GetCollItem; default;
end;
var
private
FImplementerClassCode: RawUTF8;
FName: RawUTF8;
FWBID: TSQLRecordID;
FValCol: TValueCollection;
public
constructor Create; override;
destructor Destroy; override;
published
property ImplementerClassCode:RawUTF8 index 8 read FImplementerClassCode write FImplementerClassCode;
property WorkbaseObjectID: TSQLRecordID read FWBID write FWBID stored false; //Unique!!
property Name: RawUTF8 index 50 read FName write FName;
property ValueList: TValueCollection read FValCol write FValCol;
end;
Writing the collection seems to go well, resulting in the following JSON value:
'[{"Name":"ValueName1","AttributeDef":{"DisplayName":"","Name":"ValueName1","WBAttributeClass":"TLCSSimpleValue"},"ValueObject":{"AsString":"Dit is een tekstje"}},
{"Name":"ValueName2","AttributeDef":{"DisplayName":"","Name":"ValueName2","WBAttributeClass":"TLCSSimpleValue"},"ValueObject":{"AsFloat":34.665}}]'
When the collection is read back by JSONToObject it only reads one of the items ...
Also registered as ticket: http://synopse.info/fossil/info/f99c86b1ac
Any progress on ticket 3c4146259476d06c76e5850f522ea3d093f89355?
Done
Any progress on this?
Trying to add records with an pre-filled ID, by calling
TSQLRestClient.Add(Rec, true, true);
But they are still inserted with a ID generated by a call to EngineLockedNextID.
Probable cause, the following line (MormotDB.pas, line 864) is always used when not in Batch mode
result := ExecuteFromJSON(SentData,0); // UpdatedID=0 -> insert with EngineLockedNextID
Callstack:
Mormot_D17.mORMot.TJSONObjectDecoder.Decode('"RowID":720,"ModTime":135071471049,"Name":"PietjePuck 7","Question":"{80722197-6E21-494F-A8E3-423B42A4A168}","Address_":"Parklaan 23","PostalCode":"2347BC","City":"Nuenen","Datum":"2012-05-07","FloatNumber":7,"IntNumber":2147490647,"CurrencyNumber":255.70}',(),pNonQuoted,7,True)
Mormot_D17.mORMot.TJSONObjectDecoder.Decode('{"RowID":720,"ModTime":135071471049,"Name":"PietjePuck 7","Question":"{80722197-6E21-494F-A8E3-423B42A4A168}","Address_":"Parklaan 23","PostalCode":"2347BC","City":"Nuenen","Datum":"2012-05-07","FloatNumber":7,"IntNumber":2147490647,"CurrencyNumber":255.70}',(),pNonQuoted,7,True)
Mormot_D17.mORMotDB.TSQLRestServerStaticExternal.ExecuteFromJSON('{"RowID":720,"ModTime":135071471049,"Name":"PietjePuck 7","Question":"{80722197-6E21-494F-A8E3-423B42A4A168}","Address_":"Parklaan 23","PostalCode":"2347BC","City":"Nuenen","Datum":"2012-05-07","FloatNumber":7,"IntNumber":2147490647,"CurrencyNumber":255.70}',0)
Mormot_D17.mORMotDB.TSQLRestServerStaticExternal.EngineAdd(TSQLUnitTestSampleRecord,'{"RowID":720,"ModTime":135071471049,"Name":"PietjePuck 7","Question":"{80722197-6E21-494F-A8E3-423B42A4A168}","Address_":"Parklaan 23","PostalCode":"2347BC","City":"Nuenen","Datum":"2012-05-07","FloatNumber":7,"IntNumber":2147490647,"CurrencyNumber":255.70}')
Mormot_D17.mORMot.TSQLRestServer.URI($18F318)
Mormot_D17.mORMotSQLite3.TSQLRestClientDB.InternalURI('root/UnitTestSampleRecord','POST',nil {''},$18F3F0 {''},$18F3FC {'{"RowID":720,"ModTime":135071471049,"Name":"PietjePuck 7","Question":"{80722197-6E21-494F-A8E3-423B42A4A168}","Address_":"Parklaan 23","PostalCode":"2347BC","City":"Nuenen","Datum":"2012-05-07","FloatNumber":7,"IntNumber":2147490647,"CurrencyNumber":255.70}'})
Mormot_D17.mORMot.TSQLRestClientURI.URI('root/UnitTestSampleRecord','POST',nil {''},$18F3F0 {''},$18F3FC {'{"RowID":720,"ModTime":135071471049,"Name":"PietjePuck 7","Question":"{80722197-6E21-494F-A8E3-423B42A4A168}","Address_":"Parklaan 23","PostalCode":"2347BC","City":"Nuenen","Datum":"2012-05-07","FloatNumber":7,"IntNumber":2147490647,"CurrencyNumber":255.70}'})
Mormot_D17.mORMot.TSQLRestClientURI.EngineAdd(TSQLUnitTestSampleRecord,'{"RowID":720,"ModTime":135071471049,"Name":"PietjePuck 7","Question":"{80722197-6E21-494F-A8E3-423B42A4A168}","Address_":"Parklaan 23","PostalCode":"2347BC","City":"Nuenen","Datum":"2012-05-07","FloatNumber":7,"IntNumber":2147490647,"CurrencyNumber":255.70}')
Mormot_D17.mORMot.TSQLRestClient.Add($25C4E30,True,True)
Thanks, think you forgot this one:
[dcc32 Error] SynCommons.pas(18094): E2034 Too many actual parameters
Line 18094 should be:
GetVariantFromJSON(Val,wasString,Value{$ifndef UNICODE},ForceWideString{$endif});
Updated this morning and running into all kinds of compiler errors of code segments inside {$ifdef UNICODE} directives:
[dcc32 Error] SynCommons.pas(8756): E2029 '(' expected but ')' found
[dcc32 Error] SynCommons.pas(8757): E2029 Expression expected but 'END' found
[dcc32 Error] SynCommons.pas(18094): E2034 Too many actual parameters
[dcc32 Fatal Error] mORMot_D17.dpk(38): F2063 Could not compile used unit 'SynCommons.pas'[dcc32 Error] SQLite3Commons.pas(2682): E2137 Method 'SameValue' not found in base class
[dcc32 Error] SQLite3Commons.pas(10890): E2037 Declaration of 'CompareValue' differs from previous declaration
[dcc32 Error] SQLite3Commons.pas(11041): E2037 Declaration of 'CompareValue' differs from previous declaration
[dcc32 Error] SQLite3Commons.pas(11229): E2037 Declaration of 'CompareValue' differs from previous declaration
[dcc32 Error] SQLite3Commons.pas(11310): E2037 Declaration of 'CompareValue' differs from previous declaration
[dcc32 Error] SQLite3Commons.pas(11381): E2037 Declaration of 'CompareValue' differs from previous declaration
[dcc32 Error] SQLite3Commons.pas(11574): E2037 Declaration of 'CompareValue' differs from previous declaration
[dcc32 Error] SQLite3Commons.pas(11637): E2037 Declaration of 'CompareValue' differs from previous declaration
[dcc32 Error] SQLite3Commons.pas(11683): E2037 Declaration of 'CompareValue' differs from previous declaration
[dcc32 Error] SQLite3Commons.pas(11732): E2037 Declaration of 'CompareValue' differs from previous declaration
[dcc32 Error] SQLite3Commons.pas(11781): E2037 Declaration of 'CompareValue' differs from previous declaration
[dcc32 Error] SQLite3Commons.pas(12004): E2037 Declaration of 'CompareValue' differs from previous declaration
[dcc32 Error] SQLite3Commons.pas(12120): E2037 Declaration of 'CompareValue' differs from previous declaration
[dcc32 Error] SQLite3Commons.pas(12255): E2037 Declaration of 'CompareValue' differs from previous declaration
[dcc32 Error] SQLite3Commons.pas(12358): E2037 Declaration of 'CompareValue' differs from previous declaration
I donwloaded the lastest version (2012-10-28) and running our own unittests i get the following error:
EORMException: Unhanled type for property Name
Besides the typo in "Unhanled" .. i don't understand why this exception occurs, as the published property is "just" a UTF8String that hasnt changed.
A copy of the local variables at the moment the exception is raised (TSQLPropInfoRTTI.CreateFrom(), line 19913) shown below:
Name Value
Self TSQLPropInfoRTTI
aPropInfo $3401F36
PropType $340E94C
GetProc -16777196
SetProc -16777196
StoredProc 1
Index 50
Default 0
NameIndex 1
Name 'Name'
Result ('ModTime', sftModTime, [], -2147483648)
aSQLFieldType sftAnsiText
aType $50032E24
Kind tkLString
Name 'UTF8String'
C nil
How can i use this SetWeakZero if I have a TDictionary<integer, IInterface> as i need some reference to the Interfacefield and TDictionary.AddOrSetValue....
Thanks for the fix!
Why not add a unittest for this particular problem so the test will ensure it will keep working in the future:
Synselftest.pas, line 4400
Client2 := TSQLRestClientDB.Create(Model,nil,'testpass.db3',TSQLRestServerDB,false,password);
try
Client2.Server.DB.Synchronous := smOff;
Client2.Server.DB.WALMode := true;
Client2.Server.CreateMissingTables;
Check(Client2.TransactionBegin(TSQLRecordPeople));
Check(Client2.BatchStart(TSQLRecordPeople));
Check(Client2.BatchSend(Res)=200);
Client2.Commit;
finally
Client2.Free;
end;
TSQLRestClientURI.BatchSend() really cannot cope with empty Batches, so I guess an enhancement is needed here:
Line 17810:
...
try
if fBatchCount = 0 then begin // Nothing to send
result := HTML_SUCCESS;
exit;
end;
fBatch.CancelLastComma;
...
We've created a unittest setting up a RestServer and a RestClient, when we start a batch on the client, and immediately after issue BatchSend the return value is 400 ...
procedure TLCSServerSessionTestBase.TestEmptyBatchSend;
var
Ida: TIntegerDynArray;
begin
Check(SQLWorkbaseRestClient.BatchStart(TSQLWorkbaseObject),'BatchStart failed');
CheckEquals(HTML_SUCCESS,SQLWorkbaseRestClient.BatchSend(Ida),'Empty BatchSend failed'); // FAILS, RETURN VALUE = 400
end;
This same thing happens when we use Http, NamedPipe, or Messages
Is this designed behavior? We would think that sending an empty batch is not an error, even though it is not very efficient ;-)
[Edit]
I added a new Test to TTestExternalDatabase.CryptedDatabase, and this one fails too!
Client2 := TSQLRestClientDB.Create(Model,nil,'testpass.db3',TSQLRestServerDB,false,password);
try
Client2.Server.DB.Synchronous := smOff;
Client2.Server.DB.WALMode := true;
Client2.Server.CreateMissingTables;
Check(Client2.TransactionBegin(TSQLRecordPeople));
Check(Client2.BatchStart(TSQLRecordPeople));
while False do begin
Check(R.ID<>0);
Check(Client2.BatchAdd(R,true)>=0);
end;
Check(Client2.BatchSend(Res)=200);
Client2.Commit;
finally
Client2.Free;
end;
Like i said, i registerd the needed urls in the registry with THttpApiServer.AddUrlAuthorize('root', '888', false);
but as it turns out .. i forgot the last parameter '+'
Trying to the TestSQL3 executable, registered needed urls in the registry, but i still get the error whith every assertion in test 2.5:
TSQLLite3HttpServer.Create: Impossible to register URL (administrator rights needed) for root.
I even manually changed the right in the registry where the url was registerd, but no luck so far..
What am i doing wrong???
When i'm trying to add a record to my table in MS SQL (2008) server via ODBC i get the following message
[22008] [Micrsoft][ODBC SQL Server Driver]Datetime field overflow (0)
The field involved is a Tdatetime field, created as a datetime field in the external SQL-table, and the value assigned to it is now()
callstack:
SynDBODBC.TODBCLib.HandleError(-1,3,$D1D4D0,False,sllNone)
SynDBODBC.TODBCLib.Check(-1,3,$D1D4D0,False,sllNone)
SynDBODBC.TODBCStatement.ExecutePrepared
SQLite3DB.TSQLRestServerStaticExternal.ExecuteFromJSON('{"ModTime":135058350095,"Name":"Hans Hasenack 0","Question":"{7DFAE7C2-C6F0-40D9-8D05-AA2A835A359C}","Address_":"Dit is het adres laan, 23","PostalCode":"1234AA","City":"Nijmegen","Datum":"2012-09-14T14:00:15","FloatNumber":0,"IntNumber":0,"CurrencyNumber":0}',0)
SQLite3DB.TSQLRestServerStaticExternal.EngineAdd(TSQLUnitTestSampleRecord,'{"ModTime":135058350095,"Name":"Hans Hasenack 0","Question":"{7DFAE7C2-C6F0-40D9-8D05-AA2A835A359C}","Address_":"Dit is het adres laan, 23","PostalCode":"1234AA","City":"Nijmegen","Datum":"2012-09-14T14:00:15","FloatNumber":0,"IntNumber":0,"CurrencyNumber":0}')
SQLite3Commons.TSQLRestServer.URI('root/UnitTestSampleRecord','POST','{"ModTime":135058350095,"Name":"Hans Hasenack 0","Question":"{7DFAE7C2-C6F0-40D9-8D05-AA2A835A359C}","Address_":"Dit is het adres laan, 23","PostalCode":"1234AA","City":"Nijmegen","Datum":"2012-09-14T14:00:15","FloatNumber":0,"IntNumber":0,"CurrencyNumber":0}','','',$9A164D)
SQLite3.TSQLRestClientDB.InternalURI('root/UnitTestSampleRecord','POST',$18F4BC {''},$18F568 {''},$18F574 {'{"ModTime":135058350095,"Name":"Hans Hasenack 0","Question":"{7DFAE7C2-C6F0-40D9-8D05-AA2A835A359C}","Address_":"Dit is het adres laan, 23","PostalCode":"1234AA","City":"Nijmegen","Datum":"2012-09-14T14:00:15","FloatNumber":0,"IntNumber":0,"CurrencyNumber":0}'})
SQLite3Commons.TSQLRestClientURI.URI('root/UnitTestSampleRecord','POST',nil {''},$18F568 {''},$18F574 {'{"ModTime":135058350095,"Name":"Hans Hasenack 0","Question":"{7DFAE7C2-C6F0-40D9-8D05-AA2A835A359C}","Address_":"Dit is het adres laan, 23","PostalCode":"1234AA","City":"Nijmegen","Datum":"2012-09-14T14:00:15","FloatNumber":0,"IntNumber":0,"CurrencyNumber":0}'})
SQLite3Commons.TSQLRestClientURI.EngineAdd(TSQLUnitTestSampleRecord,'{"ModTime":135058350095,"Name":"Hans Hasenack 0","Question":"{7DFAE7C2-C6F0-40D9-8D05-AA2A835A359C}","Address_":"Dit is het adres laan, 23","PostalCode":"1234AA","City":"Nijmegen","Datum":"2012-09-14T14:00:15","FloatNumber":0,"IntNumber":0,"CurrencyNumber":0}')
SQLite3Commons.TSQLRestClient.Add($104E830,True,False)
Great!! and again on such short notice!
I noticed one small breaking change:
Model.TableProps[0].SQLTableName
needs to be changed into
Model.TableProps[0].ExternalTableName
When my unittest running on Oracle auto-create the TAUTH* tables it creates the PasswordHashHexa field with length 40 (according to the index property in TSQLAuthUser)
But when adding the default users in InitializeTable() the PasswordHashHexa is longer than 40 positions, resulting in the following error:
ORA-12899: value too large for column "OPF_MORMOT"."AUTHUSER"."PASSWORDHASHHEXA" ( actual: 64. maximum 40)
The callstack being:
SynDBOracle.TSQLDBOracleLib.HandleError(-1,$54E30F8,False,sllNone)
SynDBOracle.TSQLDBOracleStatement.FetchTest(-1)
SynDBOracle.TSQLDBOracleStatement.ExecutePrepared
SQLite3DB.TSQLRestServerStaticExternal.ExecuteFromJSON('{"LogonName":"Supervisor","DisplayName":"Supervisor","PasswordHashHexa":"67aeea294e1cb515236fd7829c55ec820ef888e8e221814d24d83b3dc4d825dd","GroupRights":2}',0)
SQLite3DB.TSQLRestServerStaticExternal.EngineAdd(TSQLAuthUser,'{"LogonName":"Supervisor","DisplayName":"Supervisor","PasswordHashHexa":"67aeea294e1cb515236fd7829c55ec820ef888e8e221814d24d83b3dc4d825dd","GroupRights":2}')
SQLite3Commons.TSQLRestServer.Add($2592CC0,True,False)
SQLite3Commons.TSQLAuthGroup.InitializeTable($24E6260,'')
SQLite3.TSQLRestServerDB.CreateMissingTables(0)
Looking at the SHA256DigestToString function, it sets the length to SizeOf(TSHA256Digest) * 2
And TSHA256Digest is an array[0..31] of byte, so the result will be 64 in length
Why not add the Guest user with the same password as User, Admin, Supervisor .. that way you can't be comprising the security any more than you already do?
haha now i understand .. our MS SQL is not 2005 ... its 2000!
and the definition of nvarchar says:
nchar and nvarchar
Character data types that are either fixed-length (nchar) or variable-length (nvarchar) Unicode data and use the UNICODE UCS-2 character set.nchar(n)
Fixed-length Unicode character data of n characters. n must be a value from 1 through 4,000. Storage size is two times n bytes. The SQL-92 synonyms for nchar are national char and national character.
nvarchar(n)
Variable-length Unicode character data of n characters. n must be a value from 1 through 4,000. Storage size, in bytes, is two times the number of characters entered. The data entered can be 0 characters in length. The SQL-92 synonyms for nvarchar are national char varying and national character varying.
Remarks
When n is not specified in a data definition or variable declaration statement, the default length is 1. When n is not specified with the CAST function, the default length is 30.
Ha, i hadn't noticed the dependency ... your correct of course
[edit]
As stated in another thread, it now turn out I was using MS SQL 2000 with these tests, and not 2005 as I was thinking.
I dont know if you missed something .. but in my setup of MSSQL 2005 and using OleDB connection .. these things need to be changed to get it working..
Have you seen it working with your initial values?
When trying to run my Unittests against a MS SQL database using trhe ODBC connection, I got an "Invalid Descritor Index" error every time the external table was created by the test.
It turns out that you have to read columns in order after running a statement!
So I made the following correction to SynDBODBC.pas:
On lines 1561 and following, I orders lines so that the the ColumnUTF(x) calls were orders according to the column they were referencing, resulting in:
while Step do begin
F.ColumnName := ColumnUTF8(3);
F.ColumnType:= ODBCColumnToFieldType(ColumnInt(4),F.ColumnPrecision,F.ColumnScale);
F.ColumnTypeNative := ColumnUTF8(5);
F.ColumnLength := ColumnInt(6);
F.ColumnScale := ColumnInt(8);
F.ColumnPrecision := ColumnInt(9);
FA.Add(F);
I've created the following unittest to test if using a transaction when applying an update is working. This tests works on oracle (OCI), our own nexus driver, but now i've tried it on MS SQL using OleDB and the method stalls at line indicated in the code ...
procedure TMormotDBBaseTransactionTest.TestCommitTransaction;
var
Rec: TSQLUnitTestSampleRecord;
lStat: TSQLDBStatement;
const
cNewQuestion = 'New Question!';
cID = 2;
begin
CreateSmallModelData(Client); // Add 10 records to the UnitTestSampleRecord table
//Create another connection to test table contents outside of transaction
Conn.Connect;
Client.TransactionBegin(TSQLUnitTestSampleRecord);
Rec := TSQLUnitTestSampleRecord.CreateAndFillPrepare(Client,'ID=?', [], [cID]);
try
Rec.FillOne;
Rec.Question := cNewQuestion;
Client.Update(Rec);
// Use other connection to check that the update has not been committed yet
lStat := Conn.NewStatementPrepared('Select Question from UnittestSampleRecord where id='+ IntToString(cID), true);
try
lStat.ExecutePrepared; <<<- Here execution halts!!! No exceptions, just freezes
lStat.Step(True);
CheckNotEqualsString(Rec.Question, lStat.ColumnString(0));
finally
lStat.Free;
end;
Client.Commit;
lStat := Conn.NewStatementPrepared('Select Question from UnittestSampleRecord where id='+ IntToString(cID), true);
try
lStat.ExecutePrepared;
lStat.Step(True);
CheckEqualsString(Rec.Question, lStat.ColumnString(0));
finally
lStat.Free;
end;
finally
Rec.Free;
end;
end;
When pausign and looking at the the callstack:
:7749f8b1 ntdll.NtWaitForSingleObject + 0x15
:76151194 kernel32.WaitForSingleObjectEx + 0x43
:6fb02985 ; C:\Windows\SysWOW64\DBnmpNTw.dll
:6faaacb6 ; C:\Windows\SysWOW64\DBNETLIB.DLL
:6a4aa077 ; C:\Program Files (x86)\Common Files\System\Ole DB\sqloledb.dll
:6a47faa2 ; C:\Program Files (x86)\Common Files\System\Ole DB\sqloledb.dll
:6a47f864 ; C:\Program Files (x86)\Common Files\System\Ole DB\sqloledb.dll
:6a47f652 ; C:\Program Files (x86)\Common Files\System\Ole DB\sqloledb.dll
:6a48182f ; C:\Program Files (x86)\Common Files\System\Ole DB\sqloledb.dll
:6a4788cb ; C:\Program Files (x86)\Common Files\System\Ole DB\sqloledb.dll
:6a47f5c6 ; C:\Program Files (x86)\Common Files\System\Ole DB\sqloledb.dll
:6a6a12ff ; C:\Program Files (x86)\Common Files\System\Ole DB\oledb32.dll
SynOleDB.TOleDBStatement.ExecutePrepared
uTestMormotDBBaseCommon.TMormotDBBaseTransactionTest.TestCommitTransaction
it looks to me as if its waiting for the transaction .. while the statement shuold be excecuted in another connection (and does so with i.e. oracle OCI)
This will make it much simpler to test authentication on a default database as it adds a restricted user who restrictions can be tested immediatly
And while your fixing this .... the DB_SQLLIMITCLAUSE for MSSQL should not be top(%) but top %
Using an OleDB connection to a MS SQL server, i get an error when it tries to create the AuthUser table with the following statement:
SQLite3DB.TSQLRestServerStaticExternal.ExecuteDirect('CREATE TABLE AuthUser (ID bigint PRIMARY KEY, LogonName nvarchar(20) NOT NULL UNIQUE,DisplayName nvarchar(50),PasswordHashHexa nvarchar(40),GroupRights bigint,Data varbinary(max))',(...),(...),False)
Seems like the definition in DB_FIelds is wrong here:
(' nvarchar(%)',' bigint',' float',' money',' datetime',' nvarchar(max)',' varbinary(max)'),
I guess the "max" should be substituted with the actual maximum: 8000
If you try with an external SQLite3 properties?
How are the models created? Perhaps two tables are marked as external with diverse db and there is an issue...
What are external SQLite3 properties?
Thanks for confirming this ;-)
I do, however run into a problem that i cannot seem to fix :-( but its a little bit difficult to explain ...
I have two executables, Server and TestClient.
The server is started with the following code:
var
lModel : TLCSServerModel; //Descendant of TSQLModel, nothing new here
lConnProp: TSQLDBNexusDBConnectionProperties;
begin
lModel := TLCSServerModel.Create(BASE_MODEL_URI);
lConnProp := TSQLDBNexusDBConnectionProperties.Create('', '.\ServerDB', '', '');
VirtualTableExternalRegisterAll(lModel, lConnProp);
if not lConnProp.DatabaseExists then
lConnProp.CreateDatabase;
inherited Create(lModel, 'ServerDB.db3', True);
CreateMissingTables;
FServerModel := lModel;
FServerConnProp := lConnProp;
FRestServers := TDictionary<RawUTF8,TSQLRestServerDB>.Create;
FHttpServer := TSQLite3HttpServer.Create(cPort, [Self]);
ServiceRegister(TLCSServer,[TypeInfo(ILCSServer)],sicShared);
The server has got the following implementation of the ILCSServer interface method Opendatabase:
function TLCSServer.OpenDatabase(aConnectionString: RawUTF8; var URI: RawUTF8; aCreate: boolean = false): Boolean;
var
lConProp: TSQLDBConnectionProperties;
lRestServerDB: TSQLRestServerDB;
lModel: TLCSModel;
iCreateDB: ICanCreateDatabase;
begin
Result := False;
if aConnectionString <> '' then
begin
//Determine database type and protocol
lConProp := CreateConnectionProperties(aConnectionString);
if Supports(lConProp, ICanCreatedatabase, ICreateDB) then
if not ICreateDB.DatabaseExists then
if aCreate then
ICreateDB.CreateDatabase
else
Exit;
if lConProp <> nil then
begin
//Create separate TSQLRestSeverDB object for this database
lModel := TLCSModel.Create(URI);
VirtualTableExternalRegisterAll(lModel, lConProp);
lRestServerDB := TLCSRestDB.Create(lModel, UTF8ToString(URI) + '.db3', True);
lModel.Owner := lRestServerDB; //Make Restserver destroy model
lRestServerDB.CreateMissingTables;
//Register the new restServer with the http server, using a new URI, and add it to list of RestServers
LCSRestServer.RegisterDatabase(lRestServerDB);
Result := true;
end;
end;
end; {- TLCSServer.OpenDatabase }
Now the testclient gets started separately of course, executing the following test method:
procedure TLCSServerTest.TestOpenDatabaseAccess;
var
lDBUri: RawUTF8;
lWB: TSQLWorkbaseObject;
lClient : TSQLite3HttpClient;
lModel: TLCSModel;
const
cNewFolder = '..\..\UnitTest\NexusDBServerTest12';
begin
FModel := TLCSModel.Create(BASE_MODEL_URI);
FClient := TSQLite3HttpClient.Create('localhost','888',FModel);
CheckTrue(FClient.SetUser('Admin', 'synopse'), 'Authentication failed');
FClient.ServiceRegister([TypeInfo(ILCSServer)],sicShared);
FClient.Services[cLCSServerURI].Get(FLcsInterface);
CheckNotNull(FLcsInterface, 'LCS Interface not retrieved from server');
lDBUri := 'TestDB12';
lClient := nil;
lModel := nil;
try
CheckTrue(FLcsInterface.OpenDatabase(cNewFolder, lDBUri, True));
lModel := TLCSModel.Create(lDBUri);
lClient := TSQLite3HttpClient.Create('localhost', '888', lModel);
CheckTrue(lCLient.SetUser('Admin', 'synopse'));
lWB := TSQLWorkbaseObject.Create;
try
lWB.ID := 0;
lWB.Name := 'Blabla';
Check( lClient.Add(lWB, True) > 0, ' Adding workbaseobject failed.');
finally
lWB.Free;
end;
finally
Delete_Tree(cNewFolder);
lClient.Free;
lModel.Free;
end;
end;
If i execute this client ... it always fails with an access violation in TSQLRestServerStaticExternal.ExecuteDirect((line 1022) when it tries to execute the statement:
Query := fProperties.NewThreadSafeStatementPrepared(SQLFormat,Args,ExpectResults);
If you inspect the value of fProperties it says Inaccesible Value
This is the callstack after the access violation:
SynDB.TSQLDBConnectionProperties.NewThreadSafeStatementPrepared('select max(ID) from %',(...),True)
SQLite3DB.TSQLRestServerStaticExternal.ExecuteDirect('select max(ID) from %',(...),(...),True)
SQLite3DB.TSQLRestServerStaticExternal.EngineLockedNextID
SQLite3DB.TSQLRestServerStaticExternal.ExecuteFromJSON('{"Name":"Blabla","UserID":""}',0)
SQLite3DB.TSQLRestServerStaticExternal.EngineAdd(TSQLWorkbaseObject,'{"Name":"Blabla","UserID":""}')
SQLite3Commons.TSQLRestServer.URI('TestDB12/WorkbaseObject?session_signature=0000004C000688DA500964C9','POST','{"Name":"Blabla","UserID":""}','','',$4B8FBE7)
SQLite3HttpServer.TSQLite3HttpServer.Request('/TestDB12/WorkbaseObject?session_signature=0000004C000688DA500964C9','POST','Cache-Control: no-cache'#$D#$A'Connection: Keep-Alive'#$D#$A'Pragma: no-cache'#$D#$A'Content-Length: 29'#$D#$A'Content-Type: application/json; charset=UTF-8'#$D#$A'Accept: */*'#$D#$A'Accept-Encoding: synlz'#$D#$A'Host: localhost:888'#$D#$A'User-Agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows; FREE)'#$D#$A'RemoteIP: ::1'#$D#$A,'{"Name":"Blabla","UserID":""}','application/json; charset=UTF-8','','','')
SynCrtSock.THttpServerGeneric.Request('/TestDB12/WorkbaseObject?session_signature=0000004C000688DA500964C9','POST','Cache-Control: no-cache'#$D#$A'Connection: Keep-Alive'#$D#$A'Pragma: no-cache'#$D#$A'Content-Length: 29'#$D#$A'Content-Type: application/json; charset=UTF-8'#$D#$A'Accept: */*'#$D#$A'Accept-Encoding: synlz'#$D#$A'Host: localhost:888'#$D#$A'User-Agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows; FREE)'#$D#$A'RemoteIP: ::1'#$D#$A,'{"Name":"Blabla","UserID":""}','application/json; charset=UTF-8','','','')
SynCrtSock.THttpApiServer.Execute
Do you have any idea what i am doing wrong here ...???
No the intention was to have one server process for multiple clients
I was thinking the overall authentication would be in the model that is serviced by the main TSQLRestServerDB. This server has itrs own model, different fomr the one that are opened with the OpenDatabase() method
Well ... I had kind of a plan for our new server process that should work as follows:
The server contains a single TSQLite3HttpServer, which is created on startup, and it contains one TSQLRestServerDB. There is an interface registered to this RestServer with the following declaration:
type
ILCSServer = interface(IInvokable)
['{D54E257A-A26D-4F69-8998-4D3646137828}']
{1 open the database as specified in the connectionstring, return if this was succesfull - return the root URI of
the associated TSQLRestServer in var param URI - aConnectionstring contains all info needed to connect using
<protocol>://server/name syntax - if aCreate is true than create the database if it doent exist }
function OpenDatabase(aConnectionString: RawUTF8; var URI: RawUTF8; aCreate: boolean = false): Boolean;
function CloseDatabase(const aURI: RawUTF8): boolean;
end;
When a client connects to server and calls OpenDatabase, a new TSQLRestServerDB is created by the main server thread, and this new TSQLRestServerDB is connected to the database mentioned in the connectionstring. The new TSQLRestServerDB is also registered with the TSQLiteHttpServer so the client that opened the database, can now access this database through a newly created client? not sure here). This way a client can open multiple databases.
When the client is done with the database it should close it, hence the request for an Unregister() method.
Maybe i'm trying to create this all wrong .. if so .. please advise ....
I can use TSQLite3HttpServer.AddServer to add a newly created TRestServer to a HttpServer, but how do i remove the RestServer from TSQLite3HttpServer?
ok, thanks