You are not logged in.
Pages: 1
From my test one of my table is corrupt.
When I try to use:
DBTabellaClienti := Database.ExecuteList([], query);
to retrieve TSQLTableJSON I have some problem. The first time the TSQLTableJSON is ok but if I try to retrive it again from a client I have all black records or records with strange characters.
From my test with a new database I don't have this problem, so I think my database is corrupt.
I'd like export data from old database and import it inside the new database. Is there a way to make a dump (sql text format) to export the data and then import it? Or in alternative a way to export the data (all database or single table) as CSV and then import it?
In fact I don't think I can repair my database.
Offline
Try to execute a VACUUM command on the database (e.g. using our SynDBExplorer or any other SQLite3 editor).
See http://sqlite.org/lang_vacuum.html
If there is no "database corrupted" explicit warning, the database is not corrupted.
This is your client/server implementation which is not working as expected.
Without code to reproduce the problem (I do not speak of just some part of your code, but a code snippet easy to reproduce the issue), it is almost inpossible to find what is wrong.
Offline
I just have try VACUUM command but it not work.
I post here the code:
procedure TMainForm.AggiornaGrigliaClienti(Force: Boolean = False);
var
i, row, oldSelect: Integer;
Refreshed, Reload: Boolean;
query: string;
begin
Refreshed := False;
Reload := False;
if DBTabellaClienti = nil then
begin
query := 'SELECT Contacts.ID, Contacts.PrivateLastName, Contacts.PrivateFirstName, Contacts.WorkCompany, Contacts.PrivateAddress, Contacts.PrivateCity, ' +
'Contacts.PrivateMobile, Contacts.PrivateTelephone, Contacts.PrivateEMail, Contacts.WorkMobile, Contacts.WorkTelephone, Contacts.WorkEMail, Contacts.Role, ' +
'COUNT(ProjectsCustomers.ID) AS CPC, COUNT(ProjectsInvoices.ID) AS CPI, COUNT(ProjectsFlows.ID) AS CPF FROM Contacts LEFT JOIN ProjectsCustomers on Contacts.ID = ProjectsCustomers.IDCustomer ' +
'LEFT JOIN ProjectsInvoices on Contacts.ID = ProjectsInvoices.IDCustomer LEFT JOIN ProjectsFlows on Contacts.ID = ProjectsFlows.IDCustomer ' +
'GROUP BY Contacts.PrivateLastName, Contacts.PrivateFirstName, Contacts.WorkCompany';
DBTabellaClienti := Database.ExecuteList([], query);
Refreshed := True;
end
else
begin
if not Database.UpdateFromServer([DBTabellaClienti], Refreshed) then
begin
Refreshed := True;
Reload := True;
end;
end;
if Force then
Refreshed := True;
if Refreshed then
begin
if Reload then
begin
query := 'SELECT Contacts.ID, Contacts.PrivateLastName, Contacts.PrivateFirstName, Contacts.WorkCompany, Contacts.PrivateAddress, Contacts.PrivateCity, ' +
'Contacts.PrivateMobile, Contacts.PrivateTelephone, Contacts.PrivateEMail, Contacts.WorkMobile, Contacts.WorkTelephone, Contacts.WorkEMail, Contacts.Role, ' +
'COUNT(ProjectsCustomers.ID) AS CPC, COUNT(ProjectsInvoices.ID) AS CPI, COUNT(ProjectsFlows.ID) AS CPF FROM Contacts LEFT JOIN ProjectsCustomers on Contacts.ID = ProjectsCustomers.IDCustomer ' +
'LEFT JOIN ProjectsInvoices on Contacts.ID = ProjectsInvoices.IDCustomer LEFT JOIN ProjectsFlows on Contacts.ID = ProjectsFlows.IDCustomer ' +
'GROUP BY Contacts.PrivateLastName, Contacts.PrivateFirstName, Contacts.WorkCompany';
DBTabellaClienti := Database.ExecuteList([], query);
end;
oldSelect := -1;
if grdContacts.SelectedRow > -1 then
oldSelect := grdContacts.SelectedRow;
grdContacts.ClearRows;
row := -1;
if DBTabellaClienti <> nil then
begin
if DBTabellaClienti.RowCount > 0 then
begin
for i := 1 to DBTabellaClienti.RowCount do
begin
if ((edtRole.ItemIndex > 0) and (DBTabellaClienti.GetAsInteger(i, DBTabellaClienti.FieldIndex('Role')) = edtRole.ItemIndex - 1)) or (edtRole.ItemIndex = 0) then
begin
row := row+1;
grdContacts.AddRow(1);
if (DBTabellaClienti.GetAsInteger(i, DBTabellaClienti.FieldIndex('CPC')) + DBTabellaClienti.GetAsInteger(i, DBTabellaClienti.FieldIndex('CPI')) + DBTabellaClienti.GetAsInteger(i, DBTabellaClienti.FieldIndex('CPF'))) > 0 then
begin
grdContacts.Cell[3, row].AsInteger := (DBTabellaClienti.GetAsInteger(i, DBTabellaClienti.FieldIndex('Role')) * 2) + 1;
end
else
begin
grdContacts.Cell[3, row].AsInteger := (DBTabellaClienti.GetAsInteger(i, DBTabellaClienti.FieldIndex('Role')) * 2);
end;
grdContacts.Cell[0, row].AsInteger := DBTabellaClienti.GetAsInteger(i, DBTabellaClienti.FieldIndex('ID'));
grdContacts.Cell[4, row].AsString := DBTabellaClienti.GetString(i, DBTabellaClienti.FieldIndex('PrivateLastName'));
grdContacts.Cell[5, row].AsString := DBTabellaClienti.GetString(i, DBTabellaClienti.FieldIndex('PrivateFirstName'));
grdContacts.Cell[6, row].AsString := DBTabellaClienti.GetString(i, DBTabellaClienti.FieldIndex('WorkCompany'));
if DBTabellaClienti.GetString(i, DBTabellaClienti.FieldIndex('PrivateCity')) <> '' then
begin
grdContacts.Cell[7, row].AsString := DBTabellaClienti.GetString(i, DBTabellaClienti.FieldIndex('PrivateCity'));
end
else
begin
grdContacts.Cell[7, row].AsString := DBTabellaClienti.GetString(i, DBTabellaClienti.FieldIndex('WorkCity'));
end;
if DBTabellaClienti.GetString(i, DBTabellaClienti.FieldIndex('PrivateEMail')) <> '' then
begin
grdContacts.Cell[8, row].AsString := DBTabellaClienti.GetString(i, DBTabellaClienti.FieldIndex('PrivateEMail'));;
end
else
begin
grdContacts.Cells[8, row] := DBTabellaClienti.GetString(i, DBTabellaClienti.FieldIndex('WorkEMail'));
end;
if DBTabellaClienti.GetString(i, DBTabellaClienti.FieldIndex('PrivateMobile')) <> '' then
begin
grdContacts.Cells[9, row] := DBTabellaClienti.GetString(i, DBTabellaClienti.FieldIndex('PrivateMobile'));
end
else
begin
grdContacts.Cells[9, row] := DBTabellaClienti.GetString(i, DBTabellaClienti.FieldIndex('WorkMobile'));
end;
if DBTabellaClienti.GetString(i, DBTabellaClienti.FieldIndex('PrivateTelephone')) <> '' then
begin
grdContacts.Cells[10, row] := DBTabellaClienti.GetString(i, DBTabellaClienti.FieldIndex('PrivateTelephone'));
end
else
begin
grdContacts.Cells[10, row] := DBTabellaClienti.GetString(i, DBTabellaClienti.FieldIndex('WorkTelephone'));
end;
end;
end;
end;
end;
if (oldSelect > -1) and (oldSelect < grdContacts.RowCount) then
begin
grdContacts.SelectedRow := oldSelect;
if grdContacts.SelectedRow > -1 then
grdContacts.ScrollToRow(grdContacts.SelectedRow);
end;
btnContactEdit.Enabled := False;
btnContactDelete.Enabled := False;
end;
end;
The user can use the procedure to update the GUI (a NextGrid grid). This code works always the first time, but after the first time if I call it from the client I get the right number of records, but all records value are balck or with strange symbols (rarely). I never noticed this problem when I call the function from the server.
I use the same code also for other tables/grids and I don't have this problem.
Another information (but I think this is another problem, I don't know if you need of this), when I try to open a database (old or new) my application try to create a backup with this code:
try
Stream := TFileStream.Create(Folder + NewName + '.cbbak', fmCreate);
try
Stream.Seek(0, soFromBeginning);
DatabaseServer.Backup(Stream);
finally
Stream.Free;
end;
except
Exit;
end;
From 1/2 week the backup file have size = 0 byte and I have this error: "cannot VACUUM - SQL statements in progress".
Offline
I have make some other test when I use the code:
try
Stream := TFileStream.Create(Folder + NewName + '.cbbak', fmCreate);
try
Stream.Seek(0, soFromBeginning);
DatabaseServer.Backup(Stream);
finally
Stream.Free;
end;
except
Exit;
end;
I get alsway this error: "cannot VACUUM - SQL statements in progress". Any ideas?
Do you have find any problem on my code of post #3 about possible database corrupt?
Offline
VACUUM is working as expected - it is made several times in the regression tests.
See e.g. TTestSQLite3Engine._TSQLTableJSON
But perhaps some statements are still opened in your code.
Is it the case?
Or perhaps it is stopping because of some prepared statements in the background.
Could you try adding a fStatementCache.ReleaseAllDBStatements; call in the Backup method BEFORE the vacuum:
function TSQLRestServerDB.Backup(Dest: TStream): boolean;
var Source: TFileStream;
Closed: boolean;
begin
result := false;
if (Self=nil) or (DB=nil) then
exit;
DB.Lock(''); // flush internal cache
try
try
// VACUUM will fail if there are one or more active SQL statements
fStatementCache.ReleaseAllDBStatements;
// perform a VACCUM to recreate the database content
DB.Execute('VACUUM');
Closed := false;
try
Closed := sqlite3_close(DB.DB)=SQLITE_OK;
// compress the database content file
Source := TFileStream.Create(DB.FileName,fmOpenRead or fmShareDenyNone);
try
Dest.CopyFrom(Source,0); // Count=0 for whole stream copy
result := true;
finally
Source.Free;
end;
finally
if Closed then begin
DB.DBOpen; // reopen the database if was previously closed
InitializeEngine; // register functions and modules
end;
end;
finally
DB.UnLock;
end;
except
on E: Exception do
result := false;
end;
end;
Offline
In this moment I'm not on my office so I cannot make the test. I will do it later and then I will report the result.
In any case as you can see from post #3 the object DBTabellaClienti is not destroy. I use DBTabellaClienti and other table with UpdateFromServer, so if I destroy it I cannot use UpdateFromServer. Is this a problem?
DBTabellaClienti is create the first time when the database is open (before backup) and then is destroy when the database is closed. Until the databse is open I reuse DBTabellaClienti to know if I need update the GUI.
If this is a problem is there an alternative way?
Last edited by array81 (2012-03-13 15:37:46)
Offline
I have make a test, if I use this code:
Database := TSQLRestClientDB.Create(Model, nil, Filename, TSQLRestServerDBWithService, False);
I can backup my database without problem, but if I use this code:
Database := TSQLRestClientDB.Create(Model, nil, Filename, TSQLRestServerDBWithService, TRUE);
(aHandleUserAuthentication = TRUE), when I try to make the backup I have this error: "cannot VACUUM - SQL statements in progress". Any idea?
Besides do you have some ideas about my problem of post #3 about the possible table/database problem or curruption?
Thanks
Offline
Do you have regression tests also for backup function when aHandleUserAuthentication = TRUE?
Offline
No. There is an issue here.
I will fix it asap.
OK thanks, I will wait fot it.
I last think. I hope the last before release the beta version of my software, I continue to have problems with Database.UpdateFromServer for one of my table (see first code of post #3).
I have make a test, I hope this can help you to help me
1) Run Server and Client application;
2) Open the database and call an update GUI function, see AggiornaGrigliaClienti(Force: Boolean = False); both on server and client, all is ok. NOTE I have do a small edit of AggiornaGrigliaClienti, I use DBTabellaClienti.GetJSONValues(True); to save the DBTabellaClienti as text file. In this case I have a small file, about both on server and client computers;
3) I edit the table (on server or client), for example I change a text value field for a record (a small change);
4) Call again AggiornaGrigliaClienti both on server and client to update the GUI;
5) Result on server = right result, the GUI is update and the text files is about 96 KB (in fact I have make small change), result on client = bad result, DBTabellaClienti have the right number of rows but all records have a black value for all field, besides my text file is over 100000 KB!!! I have open the text file, there is a BIG JSON text...
I use a similar procedure with other 5 grids (same update code, only the sql is different), but I don't have this problem.
Any ideas?
Offline
I have made other test I have the same problem with a new DB, so it's not the DB the problem. I post Table struture, I hope this can help you to help me:
TSQLContacts = class(TSQLRecord)
private
fRecordCreated: TCreateTime;
fRecordModified: TModTime;
fRole: Integer;
fPrivateFirstName: RawUTF8;
fPrivateLastName: RawUTF8;
fPrivateAddress: RawUTF8;
fPrivateCity: RawUTF8;
fPrivatePostalCode: RawUTF8;
fPrivateProvince: RawUTF8;
fPrivateCountry: RawUTF8;
fPrivateBirthDate: TDateTime;
fPrivateBirthCity: RawUTF8;
fPrivateBirthCountry: RawUTF8;
fPrivateFiscalCode: RawUTF8;
fPrivateTelephone: RawUTF8;
fPrivateMobile: RawUTF8;
fPrivateFax: RawUTF8;
fPrivateEMail: RawUTF8;
fPrivateWebsite: RawUTF8;
fPrivateNetworkTwitter: RawUTF8;
fPrivateNetworkFacebook: RawUTF8;
fPrivateNetworkGoogle: RawUTF8;
fPrivateNetworkLinkedIn: RawUTF8;
fWorkCompany: RawUTF8;
fWorkJob: RawUTF8;
fWorkAddress: RawUTF8;
fWorkCity: RawUTF8;
fWorkPostalCode: RawUTF8;
fWorkProvince: RawUTF8;
fWorkCountry: RawUTF8;
fWorkVATID: RawUTF8;
fWorkTelephone: RawUTF8;
fWorkMobile: RawUTF8;
fWorkFax: RawUTF8;
fWorkEMail: RawUTF8;
fWorkWebsite: RawUTF8;
fWorkNetworkTwitter: RawUTF8;
fWorkNetworkFacebook: RawUTF8;
fWorkNetworkGoogle: RawUTF8;
fWorkNetworkLinkedIn: RawUTF8;
fNote: RawUTF8;
fDocument: TSQLRawBlob;
fDocumentExpiry: TDateTime;
fFullName: RawUTF8;
published
property RecordCreated: TCreateTime read fRecordCreated write fRecordCreated;
property RecordModified: TModTime read fRecordModified write fRecordModified;
property Role: Integer read fRole write fRole;
property PrivateFirstName: RawUTF8 read fPrivateFirstName write fPrivateFirstName;
property PrivateLastName: RawUTF8 read fPrivateLastName write fPrivateLastName;
property PrivateAddress: RawUTF8 read fPrivateAddress write fPrivateAddress;
property PrivateCity: RawUTF8 read fPrivateCity write fPrivateCity;
property PrivatePostalCode: RawUTF8 read fPrivatePostalCode write fPrivatePostalCode;
property PrivateProvince: RawUTF8 read fPrivateProvince write fPrivateProvince;
property PrivateCountry: RawUTF8 read fPrivateCountry write fPrivateCountry;
property PrivateBirthDate: TDateTime read fPrivateBirthDate write fPrivateBirthDate;
property PrivateBirthCity: RawUTF8 read fPrivateBirthCity write fPrivateBirthCity;
property PrivateBirthCountry: RawUTF8 read fPrivateBirthCountry write fPrivateBirthCountry;
property PrivateFiscalCode: RawUTF8 read fPrivateFiscalCode write fPrivateFiscalCode;
property PrivateTelephone: RawUTF8 read fPrivateTelephone write fPrivateTelephone;
property PrivateMobile: RawUTF8 read fPrivateMobile write fPrivateMobile;
property PrivateFax: RawUTF8 read fPrivateFax write fPrivateFax;
property PrivateEMail: RawUTF8 read fPrivateEMail write fPrivateEMail;
property PrivateWebsite: RawUTF8 read fPrivateWebsite write fPrivateWebsite;
property PrivateNetworkTwitter: RawUTF8 read fPrivateNetworkTwitter write fPrivateNetworkTwitter;
property PrivateNetworkFacebook: RawUTF8 read fPrivateNetworkFacebook write fPrivateNetworkFacebook;
property PrivateNetworkGoogle: RawUTF8 read fPrivateNetworkGoogle write fPrivateNetworkGoogle;
property PrivateNetworkLinkedIn: RawUTF8 read fPrivateNetworkLinkedIn write fPrivateNetworkLinkedIn;
property WorkCompany: RawUTF8 read fWorkCompany write fWorkCompany;
property WorkJob: RawUTF8 read fWorkJob write fWorkJob;
property WorkAddress: RawUTF8 read fWorkAddress write fWorkAddress;
property WorkCity: RawUTF8 read fWorkCity write fWorkCity;
property WorkPostalCode: RawUTF8 read fWorkPostalCode write fWorkPostalCode;
property WorkProvince: RawUTF8 read fWorkProvince write fWorkProvince;
property WorkCountry: RawUTF8 read fWorkCountry write fWorkCountry;
property WorkVATID: RawUTF8 read fWorkVATID write fWorkVATID;
property WorkTelephone: RawUTF8 read fWorkTelephone write fWorkTelephone;
property WorkMobile: RawUTF8 read fWorkMobile write fWorkMobile;
property WorkFax: RawUTF8 read fWorkFax write fWorkFax;
property WorkEMail: RawUTF8 read fWorkEMail write fWorkEMail;
property WorkWebsite: RawUTF8 read fWorkWebsite write fWorkWebsite;
property WorkNetworkTwitter: RawUTF8 read fWorkNetworkTwitter write fWorkNetworkTwitter;
property WorkNetworkFacebook: RawUTF8 read fWorkNetworkFacebook write fWorkNetworkFacebook;
property WorkNetworkGoogle: RawUTF8 read fWorkNetworkGoogle write fWorkNetworkGoogle;
property WorkNetworkLinkedIn: RawUTF8 read fWorkNetworkLinkedIn write fWorkNetworkLinkedIn;
property Note: RawUTF8 read fNote write fNote;
property Document: TSQLRawBlob read fDocument write fDocument;
property DocumentExpiry: TDateTime read fDocumentExpiry write fDocumentExpiry;
property FullName: RawUTF8 read fFullName write fFullName;
end;
TSQLProjectsCustomers = class(TSQLRecord)
private
fIDProject: Integer;
fIDCustomer: Integer;
fRole: RawUTF8;
fQuote: RawUTF8;
published
property IDProject: Integer read fIDProject write fIDProject;
property IDCustomer: Integer read fIDCustomer write fIDCustomer;
property Role: RawUTF8 read fRole write fRole;
property Quote: RawUTF8 read fQuote write fQuote;
end;
TSQLProjectsFlows = class(TSQLRecord)
private
fIDProject: Integer;
fIDUser: Integer;
fIDCustomer: Integer;
fIDInvoice: Integer;
fSKU: RawUTF8;
fTitle: RawUTF8;
fDate: TDateTime;
fValue: RawUTF8;
published
property IDProject: Integer read fIDProject write fIDProject;
property IDUser: Integer read fIDUser write fIDUser;
property IDCustomer: Integer read fIDCustomer write fIDCustomer;
property IDInvoice: Integer read fIDInvoice write fIDInvoice;
property SKU: RawUTF8 read fSKU write fSKU;
property Title: RawUTF8 read fTitle write fTitle;
property Date: TDateTime read fDate write fDate;
property Value: RawUTF8 read fValue write fValue;
end;
TSQLProjectsInvoices = class(TSQLRecord)
private
fIDProject: Integer;
fIDUser: Integer;
fIDCustomer: Integer;
fTitle: RawUTF8;
fDate: TDateTime;
fTotalWithdrawals: RawUTF8;
fTotalDeposits: RawUTF8;
fFee: RawUTF8;
fVATAssessment: RawUTF8;
fKind: Integer;
fNumer: RawUTF8;
fTemplate: RawUTF8;
fParameter1: Integer;
fParameter2: Integer;
fParameter3: Integer;
fParameter4: Integer;
fParameter5: Integer;
fParameterValue1: RawUTF8;
fParameterValue2: RawUTF8;
fParameterValue3: RawUTF8;
fParameterValue4: RawUTF8;
fParameterValue5: RawUTF8;
published
property IDProject: Integer read fIDProject write fIDProject;
property IDUser: Integer read fIDUser write fIDUser;
property IDCustomer: Integer read fIDCustomer write fIDCustomer;
property Title: RawUTF8 read fTitle write fTitle;
property Date: TDateTime read fDate write fDate;
property TotalWithdrawals: RawUTF8 read fTotalWithdrawals write fTotalWithdrawals;
property TotalDeposits: RawUTF8 read fTotalDeposits write fTotalDeposits;
property Fee: RawUTF8 read fFee write fFee;
property VATAssessment: RawUTF8 read fVATAssessment write fVATAssessment;
property Kind: Integer read fKind write fKind;
property Numer: RawUTF8 read fNumer write fNumer;
property Template: RawUTF8 read fTemplate write fTemplate;
property Parameter1: Integer read fParameter1 write fParameter1;
property Parameter2: Integer read fParameter2 write fParameter2;
property Parameter3: Integer read fParameter3 write fParameter3;
property Parameter4: Integer read fParameter4 write fParameter4;
property Parameter5: Integer read fParameter5 write fParameter5;
property ParameterValue1: RawUTF8 read fParameterValue1 write fParameterValue1;
property ParameterValue2: RawUTF8 read fParameterValue2 write fParameterValue2;
property ParameterValue3: RawUTF8 read fParameterValue3 write fParameterValue3;
property ParameterValue4: RawUTF8 read fParameterValue4 write fParameterValue4;
property ParameterValue5: RawUTF8 read fParameterValue5 write fParameterValue5;
end;
Offline
Two issues fixed:
- fixed VACUUM failure if there are one or more active SQL statements
- fixed potential GPF in TDynArrayHashed.ReHash after TDynArray.Clear call
Offline
I've fixed TSQLRequest.Execute(JSON: TStream) method in SynSQLite3.pas.
It now allows field names at least, even with no data (as expected by TSQLRestClientURI.UpdateFromServer).
See http://synopse.info/fossil/info/f633312bdc
Offline
Thanks, I'm testing it. At this moment I seems to work. I'll keep you updated
Offline
Ensure that you got the latest version from http://synopse.info/fossil
I've made some other fixes since the one related above.
Offline
From my test now VACUUM works but I have always the same problem with one of my table (contacts), please see post #3.
Any suggestions on what to do? I'm going crazy.
Offline
I think you are talking about the grid issue.
See http://synopse.info/forum/viewtopic.php?pid=3926#p3926
I'm working on fixing it.
Offline
Do you think is the same problem?
On my code I don't use official tdrawgrid function because I don't use a tdrawgrid on my application. I retrive the table then I update my grid (nextgrid)...
However, I hope you're right and that fix bug
Offline
I've identified and fixed an issue of unhandled buffer in TSQLTableJSON.UpdateFrom().
This was the cause of unexpected characters in TSQLTableJSON array.
Offline
Thanks. I will try it.
Offline
Pages: 1