Documentation updated to revision 1.13

ab · 2011-04-27 17:11:14

The documentation of the framework has been updated.

It's always available in its latest version in our source code repository, in the Synopse SQLite3 Framework.pro file.
Our freeware SynProject tool is needed to compile this file, together with all source code tree, into Word files.

The documentation has been generated and is available to download as pdf files from http://synopse.info/files/synproject/sampledoc.zip

The SAD file is the one to be read.
It contains the reference guide, and some tutorial to understand the framework architecture.

All the global architecture presentation (pages 28-87) has been completed, and is to be read at first.
Don't forget to find out what an ORM is for, and that you should changed your mind a little bit if you were used to use standard DB components (multi-tier, think high-level types instead of tables, stateless client-server...).

Then the main SynFile demo is analyzed in pages 469 to 481, as its General Architecture, its Database design, and its User Interface Generation (including reporting).

A lot of content has been taken from the forums, from your own questions, then completed when added to the documentation.
So please first take a look at the documentation, then ask in the forum.

I hope this may be useful!

Any feedback is welcome!

ab · 2011-05-02 06:54:59

I've updated the documentation.
An index is now available in the SAD document, to locate most keywords occurrences.

edwinsn · 2011-06-06 14:25:49

unbelievable! Very few open source projects are so well documented!

ab · 2011-06-06 15:11:23

edwinsn wrote:

unbelievable! Very few open source projects are so well documented!

But there are a lot of English mistakes in this documentation... since I'm no native!

If you can help, don't hesitate to edit the .pro file supplied in the documentation sub folder.

Thanks for your interrest.

Leander007 · 2011-06-15 12:33:28

Arnaud, is it possible to generate pdf bookmarks when you generate sampledoc pdf-s?
It would be very helpful when navigating through growing documentation.
I didn't look yet through Synopse project, so I don't know if this is easy to do automatically or not.
As I understand you don't use your pdf engine for output pdf-s, instead output from pro file is word document (doc) and then you use pdf export from Word.

Last edited by Leander007 (2011-06-15 12:34:10)

ab · 2011-06-15 13:46:01

Leander007 wrote:

Arnaud, is it possible to generate pdf bookmarks when you generate sampledoc pdf-s?

You're right, I don't use SynPdf to create the pdf yet... but SynProject creates a Word document (via a .rtf temporary file), then it relies on Word to produce the pdf.
The pdf size is huge, and... as you stated... no pdf bookmark is created, since our titles are not recognized as "titles" by Word.

You have the "Table of contents" at the beginning of the SAD document to browse it.
This is the easiest way of navigating inside the pdf.

I started to add direct pdf generation from SynProject, but it's not an easy task, because some content relies on tables and other high-level rtf markups, which need to be implemented during pdf generation.
Printing a table layout, with some formatted text or picture within, is not so easy to implement...

What I've made so far:
- Started a sub-class of our report class in SQLite3Pages to render tables (no yet finished);
- Make the rtf generation a class, with some high-level methods, able to produce some other content.

Using rtf and Word as generator allows to change the rendered content just before printing / published as pdf, for instance by adding some page breaks.

I'd like to create some html content also, in order to produce the documentation as web site. It could be great to have the whole documentation of the framework "browsable" from our web site, won't it?

So perhaps SynProject could create html content:
- Either in multiple pages to produce an online web site ready to be published on Internet;
- Either a big one page HTML file, which may be rendered via THtmlView into a pdf document.

So perhaps the HTML solution is better than a SQLite3Pages renderer. It could produce both web site and pdf file from the same code base.

Still a work in progress... any idea is welcome!

Leander007 · 2011-06-15 14:28:33

Yes, HTML would be better and user editable...
Maybe even better as custom XML (raw data) with some XSLT transformation to get HTML or some other output.

Leander007 · 2011-06-15 15:05:51

Maybe is even more reasonable to take XML approach as from Delphi 2006 there is option to generate XML documentation (same as in .NET) and could be used for extraction for code comments in future and could be combined with other XML output. What is really nice if you have some cross links (e.g. see, seealso ...) which points to some non existing type, you get warning on compilation.

Ok, if the comments in code are converted to the xml doc format, then documentation from code could not be generated with Delphi prior 2006 (maybe Doc-O-Matic Express...) with Delphi itself.

This are just some thoughts, because I already use (still experimental) in some projects Doc-O-Matic Express for xml doc conversion to the chm (html) help, but I want to work directly with Delphi xml doc and having some custom logic (as you did with Synopse) which works with "static" help pages or even with wiki approach.

With CnPack or GExperts (it's peaty that this auto commenting features are not included in Delphi by default as it is in Visual Studio) is very easy to quickly comment units, classes, procedures. I'm using regions to expand or collapse comments, that is very handy too.

{$REGION 'xmldoc'}
/// <summary>
/// Token importance degree. 0 is for informative message, 1 is alarm and so on.
/// </summary>
{$ENDREGION}
property AlarmType: integer read _alarmType write _alarmType;

Last edited by Leander007 (2011-06-15 15:12:51)

ab · 2011-06-15 15:13:29

All comments are already extracted from the source code by our own parser included in SynProject.

I found out that the XML doc format is really verbose and does pollute the code with XML tags.
We use our own .pro format, which uses '-' chars to start a description item, rendered as a list in the documentation, and allow to put source code in the comments for lines beginning with # or $, is already very good.

And the general layout of our SynProject documentation is much more elaborated than any other existing xml doc format.
You have a whole documentation workflow, from Design Input and Risk Analysis, Architecture, Design, Test, Release notes, and more, included in one source text file. All code logic and declarations are also included in the SAD document, with a lot of features, like GraphViz diagrams, cross-reference tables (including bug tracking or ISO requirements...).

Take a look at http://synopse.info/fossil/wiki?name=SynProject and you'll find out how SynProject is more advanced than any other documentation process I know about.

Leander007 · 2011-06-15 23:26:42

By mistake I named SynProject as Synopse in my last post. I'm aware (from your description, not really used it) of your completeness of this documenting program.

Maybe is not totally proper that I opened different but somehow connected subject, when I mentioned XML (with XSLT you have the power needed) as possible output for documentation and at the same time I was talking about XML DOC comments, which are standard at least in Microsoft world.

I really like xml doc comments, because I don't take this kind of xml as evil (opposite from that I see SOAP as too verbose). If you have proper tools at your hand this is not the obstacle. With tools I mean, using regions to hide comments, auto generated templates (automatically extracted arguments...) for procedures and functions for example, help insight showing your comments in Delphi IDE, Delphi checks referencing comments integrity when compiling and I can bet that some tool is capable to refactor referencing comments when you e.g. rename class.

If I understand correctly you use modified PasDoc (as I know all old parsers brake on new features of Delphi language, so constant update of them is the must) for comment extraction and then you generate pro file.
As I see it, xml doc could be replacement (or the extra possibility) only for PasDoc parser (not as the whole) as intermediate format, which has well formed xml format and could be easily parsed and transformed into pro format (or could be used for totally different cases, because it is standardized...).

In short, if I want to comment code in xml doc and use the generated Delphi xml documentation output as source for SynProject, which parts need the change? Is it only the PasDoc parser or is the "picture" more complex?
If this is not too hard, maybe I can try and add support for Delphi generated XML documentation, which is the code (with or without comments) extraction only.

Last edited by Leander007 (2011-06-15 23:27:30)

ab · 2011-06-16 05:22:46

I guess than only the PasDoc parser should be replaced by xml doc.
It's used to retrieve all types, functions, classes and constants declarations, and also retrieve the comments.
You'll have to change the comments format, to add the <xml> tags within.
But it could be very easy to implement it. The TProjectParser class is a stand-alone class (SynProject can even be used without any parser).
One good point of XML doc is that it could be used to integrate DotNet source code. I'd like to be able to do that in the next future.
So, of course, your idea is not wrong, if it meant to replace the PasDoc parser (which won't be able to parse latest Delphi code, e.g. generics).

Leander007 · 2011-06-16 09:42:37

Yes, that was my point, only replacing PasDoc parser with xml doc parser, what Delphi itself is already. So if they add some new feature to Pascal language, we don't feel this, because xml doc output format is consistent and parsing xml is very easy.

Don't forget not only that you are compatible with .NET commenting, but you get Help Insight, compiler warnings about wrong referenced "classes" and maybe refactoring tools too. In short Embercadero is enforcing Microsoft standards and they (both) make tools for us.

I think would be best to use the Delphi generated xml output and parse it to some intermediate format which would fit in SynProject.
This way we don't need custom parser implementation for xml comments in code, rather we parse the well formated generated xml from Delphi compiler.
In this way you have generated xml with all classes, functions, records even if you did not comment anything. You get all hierarchy and you could easily generate final documentation only for "elements" which are commented if you wish.

What you think about that?

Last edited by Leander007 (2012-01-07 23:32:16)

ab · 2011-06-16 12:04:04

With PasDoc, you have already all interface information even if you did not comment anything.

But I don't think this is a good idea to comment nothing, or comment with some redundant information (e.g. "this method AddToList expect one Item parameter which is added to the list"). When commenting the framework, I tried to add unique information, like links to other methods, source code sample, and some kind of specifications. In fact, I first write the interface and their comments as associated specification, then write the test, then write the implementation.

Truly instructive comments are mandatory, IMHO.
Without such comments, we can't call it documentation. In this case, source code and IDE works better than a plain document with interfaces.

Leander007 · 2011-06-16 13:22:16

Quick look through comments in your code give me the filling of only <summary> node usage and missing others (as see, seealso, param...), which are used in chm, MS Help 2, HTML and other help formats used as API description in Delphi IDE and similar.

E.g. automatically inserted template (with regions of course) with typing almost only ///:

{$REGION 'xmldoc'}
/// <summary>
/// Register a specified test to this class instance
/// </summary>
/// <param name="aMethod">Comment here the aMethod param</param>
/// <param name="aName">Comment here the aName param</param>
{$ENDREGION}
procedure Add(aMethod: TSynTestEvent; const aName: string);

with excerpt of generated xml doc (which is really verbose, too verbose, but quickly generated by compiler and you can guess that all xml tags are not completely the same as in .NET, but this is not the problem):

(...)
<procedure name="Add" visibility="public">
<devnotes>
<summary>
 Register a specified test to this class instance 
</summary>
<param name="aMethod">
Comment here the aMethod param
</param>
<param name="aName">
Comment here the aName param
</param>
</devnotes>
<parameters>
<parameter name="aMethod" type="TSynTestEvent"/>
<parameter name="aName" type="string" paramflags="const"/>
</parameters>
</procedure>
(...)

versus

/// register a specified test to this class instance
procedure Add(aMethod: TSynTestEvent; const aName: string);

Xml Doc is indeed more verbose (with more information in this case) when used for commenting the code, but with using regions that is not any problem, because they are collapsible and you don't see this as nagging text.

-------------
I saw that you have some special tags in comments, but I don't know if they could achieve same effect as regular xml doc tags?
Are this comment tags used in your code described anywhere?
In SynProject you have some kind of version control, but is it possible (with minor work) to attach some 3rd version control, such as subversion or fossil ?

I'll definitely look at SynProject in details to become more familiar with used concepts. Then I can properly make a comment, what could be done to easily interface (or extend) with existing tools, practices...

Last edited by Leander007 (2011-06-16 13:31:04)

ab · 2011-06-16 13:57:08

The only more information expected by xml doc are the parameters.
I found out that the parameter names should be self-explanatory about their meaning, most of the time.
The framework documentation has the "summary" as its first paragraphs, then a list of details, each one with lines starting with a '-' (minus sign).
For instance:

/// register a specified test to this class instance
// - aMethod should refers to an event handler implementing a test
// - aName should be unique, i.e. not already used by previous registered methods
procedure Add(aMethod: TSynTestEvent; const aName: string);

And from my experiment, some parameters are sometimes not individuals. They are used in diverse ways, depending of the calling context. In this case, I found out that a plain list of "-" cases in plain English is always more complete and efficient than a per-parameter description.

The documentation makes exhaustive usage of this convension.
For instance:

/// guess the content type of an UTF-8 encoded field value, as used in TSQLTable.Get()
// - if P if nil or 'null', return sftUnknown
// - otherwize, guess its type from its value characters
// - sftBlob is returned if the field is encoded as SQLite3 BLOB literals
// (X'53514C697465' e.g.) or with '\uFFF0' magic code
// - since P is PUTF8Char, string type is sftUTF8Text only
// - sftFloat is returned for any floating point value, even if it was
// declared as sftCurrency type
// - sftInteger is returned for any INTEGER stored value, even if it was declared
// as sftEnumerate, sftSet, sftID, sftRecord, sftBoolean or sftTimeLog type
function UTF8ContentType(P: PUTF8Char): TSQLFieldType;

The "see also" and such references are replaced by more efficient plain text presentation (the parser tries to discover the name of other properties and methods in the text), and are not so necessary when you have the whole source code. Modern IDE (e.g. the Ctrl+Click on a work) makes wonders.
But there is room from improvements here, you're right.

I was also always convinced than a plain code declaration like

procedure Add(aMethod: TSynTestEvent; const aName: string);

makes always more sense than such a list:

<parameters>
<parameter name="aMethod" type="TSynTestEvent"/>
<parameter name="aName" type="string" paramflags="const"/>
</parameters>

in the final documentation.
Of course, it depends on how the final document use this <parameters> nodes, but I don't think of any benefit about not showing the interface as written in the code. It's always more concise and exhaustive, at least for a library documentation (readers should understant the library language conventions, and are usually not SOAP-like machine expecting parameters one by one).

But you are right.
There are a lot of improvements to be made to SynProject.
Using XML Doc as an alternate source of parsing data should be added.

mORMot Open Source

#1 2011-04-27 17:11:14

Documentation updated to revision 1.13

#2 2011-05-02 06:54:59

Re: Documentation updated to revision 1.13

#3 2011-06-06 14:25:49

Re: Documentation updated to revision 1.13

#4 2011-06-06 15:11:23

Re: Documentation updated to revision 1.13

#5 2011-06-15 12:33:28

Re: Documentation updated to revision 1.13

#6 2011-06-15 13:46:01

Re: Documentation updated to revision 1.13

#7 2011-06-15 14:28:33

Re: Documentation updated to revision 1.13

#8 2011-06-15 15:05:51

Re: Documentation updated to revision 1.13

#9 2011-06-15 15:13:29

Re: Documentation updated to revision 1.13

#10 2011-06-15 23:26:42

Re: Documentation updated to revision 1.13

#11 2011-06-16 05:22:46

Re: Documentation updated to revision 1.13

#12 2011-06-16 09:42:37

Re: Documentation updated to revision 1.13

#13 2011-06-16 12:04:04

Re: Documentation updated to revision 1.13

#14 2011-06-16 13:22:16

Re: Documentation updated to revision 1.13

#15 2011-06-16 13:57:08

Re: Documentation updated to revision 1.13

Board footer