logo.png
mORMot2 API Reference

mormot.core.buffers.pas unit

Purpose: Framework Core Low-Level Memory Buffer Process
- this unit is a part of the Open Source Synopse mORMot framework 2, licensed under a MPL/GPL/LGPL three license - see LICENSE.md

1.1. Units used in the mormot.core.buffers unit

Unit NameDescription
mormot.core.baseFramework Core Shared Types and RTL-like Functions
mormot.core.osFramework Core Low-Level Wrappers to the Operating-System API
mormot.core.rttiFramework Core Low-Level Cross-Compiler RTTI Definitions
mormot.core.textFramework Core Low-Level Text Processing
mormot.core.unicodeFramework Core Low-Level Unicode UTF-8 UTF-16 Ansi Conversion

1.2. mormot.core.buffers class hierarchy

TSynPersistentTMemoryMapTextTStreamWithPositionAndSizeTNestedStreamReaderTBufferedStreamReaderTStreamRedirectHash32TStreamRedirectCrc32cTStreamRedirectTStreamWithPositionTStreamTFakeWriterStreamTObjectTRawByteStringHeapTRawByteStringGroupTRawByteStringBufferTProgressInfoTFastReaderTBufferWriterTAlgoCompressTAlgoSynLZTAlgoCompressWithNoDestLenTAlgoRleLZTAlgoRleESynExceptionEStreamRedirectEFastReaderEBufferExceptionEAlgoCompress
mormot.core.buffers class hierarchy

1.3. Objects implemented in the mormot.core.buffers unit

ObjectsDescription
EAlgoCompressException raised by TAlgoCompress classes
EBufferExceptionException raised during buffer processing
EFastReaderException raised during TFastReader decoding
EStreamRedirectException raised during TStreamRedirect processing
TAlgoCompressAbstract low-level parent class for generic compression/decompression algorithms
TAlgoCompressWithNoDestLenAbstract class storing the plain length before calling compression API
TAlgoRleImplement RLE compression as a TAlgoCompress class
TAlgoRleLZImplement our SynLZ compression with RLE preprocess as a TAlgoCompress class
TAlgoSynLZImplement our fast SynLZ compression as a TAlgoCompress class
TBufferedStreamReaderTStream with an internal memory buffer
TBufferWriterThis class can be used to speed up writing to a file or a stream
TFakeWriterStreamA fake TStream, which will just count the number of bytes written
TFastReaderSafe decoding of a TBufferWriter content from an in-memory buffer
TMemoryMapTextAble to read a UTF-8 text file using memory map
TMultiPartUsed by MultiPartFormDataDecode() to return one item of its data
TNestedStreamReaderTStream allowing to read from some nested TStream instances
TProgressInfoInformation about the progression of a process, e.g. for TStreamRedirect
TRawByteStringBufferStore one RawByteString content with an associated length
TRawByteStringGroupStore several RawByteString content with optional concatenation
TRawByteStringGroupValueItem as stored in a TRawByteStringGroup instance
TRawByteStringHeapThread-safe reusable set of constant RawByteString/RawUtf8 instances
TStreamRedirectAn abstract pipeline stream able to redirect and hash read/written content
TStreamRedirectCrc32cTStreamRedirect with crc32c 32-bit checksum
TStreamRedirectHash32TStreamRedirect with 32-bit THasher checksum
TValueResultKind of result returned by FromVarBlob() function

1.3.1. TValueResult

TValueResult = record

Kind of result returned by FromVarBlob() function


Len: PtrInt;

Value length (in bytes)


Ptr: PAnsiChar;

Start of data value


1.3.2. EAlgoCompress

EAlgoCompress = class(ESynException)

Exception raised by TAlgoCompress classes


1.3.3. TAlgoCompress

TAlgoCompress = class(TObject)

Abstract low-level parent class for generic compression/decompression algorithms
- will encapsulate the compression algorithm with crc32c hashing
- all Algo* abstract methods should be overriden by inherited classes
- don't inherit from TSynPersistent since we don't need any of it


constructor Create; virtual;

Will register AlgoID in the global list, for Algo() class methods
- no need to free this instance, since it will be owned by the global list
- raise a EAlgoCompress if the class or its AlgoID are already registered
- you should never have to call this constructor, but define a global variable holding a reference to a shared instance


destructor Destroy; override;

Finalize this algorithm


class function Algo(Comp: PAnsiChar; CompLen: integer): TAlgoCompress; overload;

Get the TAlgoCompress instance corresponding to the AlgoID stored in the supplied compressed buffer
- returns nil if no algorithm was identified


class function Algo(const Comp: RawByteString): TAlgoCompress; overload;

Get the TAlgoCompress instance corresponding to the AlgoID stored in the supplied compressed buffer
- returns nil if no algorithm was identified


class function Algo(const Comp: TByteDynArray): TAlgoCompress; overload;

Get the TAlgoCompress instance corresponding to the AlgoID stored in the supplied compressed buffer
- returns nil if no algorithm was identified


class function Algo(aAlgoID: byte): TAlgoCompress; overload;

Get the TAlgoCompress instance corresponding to the supplied AlgoID
- returns nil if no algorithm was identified
- stored content is identified as TAlgoSynLZ


class function Algo(Comp: PAnsiChar; CompLen: integer; out IsStored: boolean): TAlgoCompress; overload;

Get the TAlgoCompress instance corresponding to the AlgoID stored in the supplied compressed buffer
- returns nil if no algorithm was identified
- also identifies "stored" content in IsStored variable


function AlgoCompress(Plain: pointer; PlainLen: integer; Comp: pointer): integer; virtual; abstract;

This method will compress the supplied data


function AlgoCompressDestLen(PlainLen: integer): integer; virtual; abstract;

Get maximum possible (worse) compressed size for the supplied length


function AlgoDecompress(Comp: pointer; CompLen: integer; Plain: pointer): integer; virtual; abstract;

This method will decompress the supplied data


function AlgoDecompressDestLen(Comp: pointer): integer; virtual; abstract;

This method will return the size of the decompressed data


function AlgoDecompressPartial(Comp: pointer; CompLen: integer; Partial: pointer; PartialLen, PartialLenMax: integer): integer; virtual; abstract;

This method will partially and safely decompress the supplied data
- expects PartialLen <= result < PartialLenMax, depending on the algorithm


function AlgoHash(ForceHash32: boolean; Data: pointer; DataLen: integer): cardinal; overload;

Computes the digital signature of the buffer, or Hash32() if specified


function AlgoHash(Previous: cardinal; Data: pointer; DataLen: integer): cardinal; overload; virtual;

Computes by default the crc32c() digital signature of the buffer


function AlgoName: TShort16;

Returns the algorithm name, from its classname
- e.g. TAlgoSynLZ->'synlz' TAlgoLizard->'lizard' nil->'none' TAlgoDeflateFast->'deflatefast'


function Compress(Plain: PAnsiChar; PlainLen: integer; CompressionSizeTrigger: integer = 100; CheckMagicForCompressed: boolean = false; BufferOffset: integer = 0): RawByteString; overload; virtual;

Compress a memory buffer with crc32c hashing to a RawByteString


function Compress(const Plain: RawByteString; CompressionSizeTrigger: integer = 100; CheckMagicForCompressed: boolean = false; BufferOffset: integer = 0): RawByteString; overload;

Compress a memory buffer with crc32c hashing to a RawByteString


function Compress(Plain, Comp: PAnsiChar; PlainLen, CompLen: integer; CompressionSizeTrigger: integer = 100; CheckMagicForCompressed: boolean = false): integer; overload; virtual;

Compress a memory buffer with crc32c hashing
- supplied Comp buffer should contain at least CompressDestLen(PlainLen) bytes


function CompressDestLen(PlainLen: integer): integer;

Get maximum possible (worse) compressed size for the supplied length
- including the crc32c + algo 9 bytes header


function CompressToBytes(Plain: PAnsiChar; PlainLen: integer; CompressionSizeTrigger: integer = 100; CheckMagicForCompressed: boolean = false): TByteDynArray; overload;

Compress a memory buffer with crc32c hashing to a TByteDynArray


function CompressToBytes(const Plain: RawByteString; CompressionSizeTrigger: integer = 100; CheckMagicForCompressed: boolean = false): TByteDynArray; overload;

Compress a memory buffer with crc32c hashing to a TByteDynArray


function Decompress(const Comp: TByteDynArray): RawByteString; overload;

Uncompress a RawByteString memory buffer with crc32c hashing


function Decompress(const Comp: RawByteString; out PlainLen: integer; var tmp: RawByteString; Load: TAlgoCompressLoad = aclNormal): pointer; overload;

Uncompress a RawByteString memory buffer with crc32c hashing
- returns nil if crc32 hash failed, i.e. if the supplied Comp is not correct
- returns a pointer to the uncompressed data and fill PlainLen variable, after crc32c hash
- avoid any memory allocation in case of a stored content - otherwise, would uncompress to the tmp variable, and return pointer(tmp) and length(tmp)


function Decompress(Comp: PAnsiChar; CompLen: integer; out PlainLen: integer; var tmp: RawByteString; Load: TAlgoCompressLoad = aclNormal): pointer; overload;

Uncompress a RawByteString memory buffer with crc32c hashing
- returns nil if crc32 hash failed, i.e. if the supplied Data is not correct
- returns a pointer to an uncompressed data buffer of PlainLen bytes
- avoid any memory allocation in case of a stored content - otherwise, would uncompress to the tmp variable, and return pointer(tmp) and length(tmp)


function Decompress(const Comp: RawByteString; Load: TAlgoCompressLoad = aclNormal; BufferOffset: integer = 0): RawByteString; overload;

Uncompress a RawByteString memory buffer with crc32c hashing


function DecompressBody(Comp, Plain: PAnsiChar; CompLen, PlainLen: integer; Load: TAlgoCompressLoad = aclNormal): boolean; virtual;

Decode the content of a memory buffer compressed via the Compress() method
- PlainLen has been returned by a previous call to DecompressHeader()


function DecompressHeader(Comp: PAnsiChar; CompLen: integer; Load: TAlgoCompressLoad = aclNormal): integer; virtual;

Decode the header of a memory buffer compressed via the Compress() method
- validates the crc32c of the compressed data (unless Load=aclNoCrcFast), then return the uncompressed size in bytes, or 0 if the crc32c does not match
- should call DecompressBody() later on to actually retrieve the content


function DecompressPartial(Comp, Partial: PAnsiChar; CompLen, PartialLen, PartialLenMax: integer): integer; virtual;

Partial decoding of a memory buffer compressed via the Compress() method
- returns 0 on error, or how many bytes have been written to Partial
- will call virtual AlgoDecompressPartial() which is slower, but expected to avoid any buffer overflow on the Partial destination buffer
- some algorithms (e.g. Lizard) may need some additional bytes in the decode buffer, so PartialLenMax bytes should be allocated in Partial^, with PartialLenMax > expected PartialLen, and returned bytes may be > PartialLen, but always <= PartialLenMax


function FileCompress(const Source, Dest: TFileName; Magic: cardinal; ForceHash32: boolean = false; ChunkBytes: Int64 = 128 shl 20; WithTrailer: boolean = false): boolean; virtual;

Compress a file content using this compression algorithm
- source file is split into ChunkBytes blocks (128 MB by default) for fast in-memory compression of any file size, then compressed and including checksums of Source/Dest data
- it is not compatible with StreamCompress format, which has no chunking
- you should specify a Magic number to be used to identify the compressed file format
- follow the FileSynLZ() deprecated function format, if ForceHash32=true so that Hash32() is used instead of the AlgoHash() of this instance
- may be overriden to support a standard file layout (e.g. AlgoGZ)


class function FileIsCompressed(const Name: TFileName; Magic: cardinal): boolean; virtual;

Returns TRUE if the supplied file name is a compressed file, matching the Magic number as supplied to FileCompress() function
- follow the FileIsSynLZ() deprecated function format
- expects the compressed data to be at file beginning (not appended)
- may be overriden to support a standard file layout (e.g. AlgoGZ)


function FileUnCompress(const Source, Dest: TFileName; Magic: cardinal; ForceHash32: boolean = false): boolean; virtual;

Uncompress a file previously compressed via FileCompress()
- you should specify a Magic number to be used to identify the compressed file format
- follow the FileUnSynLZ() deprecated function format, if ForceHash32=true so that Hash32() is used instead of the AlgoHash() of this instance
- may be overriden to support a standard file layout (e.g. AlgoGZ)


function StreamCompress(Source: TStream; const DestFile: TFileName; Magic: cardinal; ForceHash32: boolean = false; WithTrailer: boolean = true; ChunkBytes: PtrInt = 4 shl 20): Int64; overload;

Compress a Stream content using this compression algorithm into a file
- just a wrapper around the overloaded StreamCompress() method


function StreamCompress(Source, Dest: TStream; Magic: cardinal; ForceHash32: boolean = false; WithTrailer: boolean = true; ChunkBytes: PtrInt = 4 shl 20): Int64; overload;

Compress a Stream content using this compression algorithm
- source Stream may be read and compressed by ChunkBytes = 4MB chunks
- a 32-bit Magic number identifies the compressed content chunks
- WithTrailer would finish the Dest output with a trailer block to locate the position of the compressed data, to be used e.g. when it is appended
- follow the StreamSynLZ() deprecated function format, if ForceHash32=true and WithTrailer=true so that Hash32() is used instead of AlgoHash()


function StreamComputeLen(P: PAnsiChar; Len: PtrUInt; Magic: cardinal): integer;

Compute the length of a given StreamCompress() buffer from its trailer
- allows to replace an existing appended content, for instance
- expects StreamCompress(WithTrailer=true) format


function StreamUnCompress(Source, Dest: TStream; Magic: cardinal; ForceHash32: boolean = false): boolean; overload;

Uncompress a Stream previously compressed via StreamCompress()
- you should specify a Magic number to be used to identify the compressed Stream format
- if Source is not positioned at compressed data beginning, a trailer is searched at the end of the Source stream to get the proper location
- returns true on success, and false on decoding error - but some chunks may have been decompressed in Dest even if false is returned


function StreamUnCompress(const Source: TFileName; Magic: cardinal; ForceHash32: boolean = false): TMemoryStream; overload;

Uncompress a File previously compressed via StreamCompress() as TStream
- you should specify a Magic number to be used to identify the compressed Stream format
- follow the StreamUnSynLZ() deprecated function format, if ForceHash32=true so that Hash32() is used instead of the AlgoHash() of this instance
- if the compressed data is not at Source file beginning, a trailer is searched at the end of the Source content to get the proper location


function StreamUnCompress(Source: TStream; Magic: cardinal; ForceHash32: boolean = false): TMemoryStream; overload;

Uncompress a Stream previously compressed via StreamCompress()
- return nil on decompression error, or a new TMemoryStream instance
- follow the StreamUnSynLZ() deprecated function format, if ForceHash32=true so that Hash32() is used instead of the AlgoHash() of this instance


function TryDecompress(const Comp: RawByteString; out Dest: RawByteString; Load: TAlgoCompressLoad = aclNormal): boolean;

Uncompress a RawByteString memory buffer with crc32c hashing
- returns TRUE on success


class function UncompressedSize(const Comp: RawByteString): integer;

Quickly validate a compressed buffer content, without uncompression
- extract the TAlgoCompress, and call DecompressHeader() to check the hash of the compressed data, and return then uncompressed size
- returns 0 on error (e.g. unknown algorithm or incorrect hash)


procedure Decompress(Comp: PAnsiChar; CompLen: integer; out result: RawByteString; Load: TAlgoCompressLoad = aclNormal; BufferOffset: integer = 0); overload;

Uncompress a memory buffer with crc32c hashing


property AlgoFileExt: TFileName read fAlgoFileExt;

The usual file extension of this algorithm
- e.g. '.synlz' or '.synz' or '.synliz' for SynLZ, Deflate or Lizard


property AlgoHasForcedFormat: boolean read fAlgoHasForcedFormat;

If this algorithm does not supports our custom storage format
- e.g. AlgoGZ set true and only supports plain buffers and files methods and would raise EAlgoCompress when stream methods are used


property AlgoID: byte read fAlgoID;

Contains a genuine byte identifier for this algorithm
- 0 is reserved for stored, 1 for TAlgoSynLz, 2/3 for TAlgoDeflate/Fast (in mormot.core.zip.pas), 4/5/6 for TAlgoLizard/Fast/Huffman (in mormot.lib.lizard.pas), 7/8 for TAlgoRleLZ/TAlgoRle, 9/10 for limited TAlgoGZ/TAlgoGZFast (in mormot.core.zip.pas)


1.3.4. TAlgoSynLZ

TAlgoSynLZ = class(TAlgoCompress)

Implement our fast SynLZ compression as a TAlgoCompress class
- please use the AlgoSynLZ global variable methods instead of the deprecated SynLZCompress/SynLZDecompress wrapper functions


constructor Create; override;

Set AlgoID = 1 as genuine byte identifier for SynLZ


function AlgoCompress(Plain: pointer; PlainLen: integer; Comp: pointer): integer; override;

Compress the supplied data using SynLZ


function AlgoCompressDestLen(PlainLen: integer): integer; override;

Get maximum possible (worse) SynLZ compressed size for the supplied length


function AlgoDecompress(Comp: pointer; CompLen: integer; Plain: pointer): integer; override;

Decompress the supplied data using SynLZ


function AlgoDecompressDestLen(Comp: pointer): integer; override;

Return the size of the SynLZ decompressed data


function AlgoDecompressPartial(Comp: pointer; CompLen: integer; Partial: pointer; PartialLen, PartialLenMax: integer): integer; override;

Partial (and safe) decompression of the supplied data using SynLZ


1.3.5. TAlgoCompressWithNoDestLen

TAlgoCompressWithNoDestLen = class(TAlgoCompress)

Abstract class storing the plain length before calling compression API
- some libraries (e.g. Deflate or Lizard) don't provide the uncompressed length from its output buffer - inherit from this class to store this value as ToVarUInt32, and override the RawProcess abstract protected method


function AlgoCompress(Plain: pointer; PlainLen: integer; Comp: pointer): integer; override;

Performs the compression, storing PlainLen and calling protected RawProcess


function AlgoDecompress(Comp: pointer; CompLen: integer; Plain: pointer): integer; override;

Performs the decompression, retrieving PlainLen and calling protected RawProcess


function AlgoDecompressDestLen(Comp: pointer): integer; override;

Return the size of the decompressed data (using FromVarUInt32)


function AlgoDecompressPartial(Comp: pointer; CompLen: integer; Partial: pointer; PartialLen, PartialLenMax: integer): integer; override;

Performs the decompression, retrieving PlainLen and calling protected RawProcess


1.3.6. TAlgoRleLZ

TAlgoRleLZ = class(TAlgoCompressWithNoDestLen)

Implement our SynLZ compression with RLE preprocess as a TAlgoCompress class
- SynLZ is very good with JSON or text, but not so efficient when its input has a lot of padding (e.g. a database file, or unoptimized raw binary)
- this class would make a first pass with RleCompress() before SynLZ
- if RLE has no effect during compression, will fallback to plain SynLZ with no RLE pass during decompression


constructor Create; override;

Set AlgoID = 7 as genuine byte identifier for RLE + SynLZ


function AlgoCompressDestLen(PlainLen: integer): integer; override;

Get maximum possible (worse) compressed size for the supplied length


1.3.7. TAlgoRle

TAlgoRle = class(TAlgoCompressWithNoDestLen)

Implement RLE compression as a TAlgoCompress class
- if RLE has no effect during compression, will fallback to plain store


constructor Create; override;

Set AlgoID = 8 as genuine byte identifier for RLE


function AlgoCompressDestLen(PlainLen: integer): integer; override;

Get maximum possible (worse) compressed size for the supplied length


1.3.8. EFastReader

EFastReader = class(ESynException)

Exception raised during TFastReader decoding


1.3.9. TFastReader

TFastReader = object(TObject)

Safe decoding of a TBufferWriter content from an in-memory buffer
- raise a EFastReader exception on decoding error (e.g. if a buffer overflow may occur) or call OnErrorOverflow/OnErrorData event handlers


CustomVariants: pointer;

When used to unserialize variants, stores options for TDocVariant creation
- contains a PDocVariantOptions reference pointer as defined in the mormot.core.data unit


Last: PAnsiChar;

The last position in the buffer


OnErrorData: TOnFastReaderErrorData;

Use this event to customize the ErrorData process


OnErrorOverflow: TOnFastReaderErrorOverflow;

Use this event to customize the ErrorOverflow process


P: PAnsiChar;

The current position in the memory


Tag: PtrInt;

Some opaque value, e.g. a version number to define the binary layout


function CopySafe(Dest: pointer; DataLen: PtrInt): boolean;

Copy data from the current position, and move ahead the specified bytes
- this version won't call ErrorOverflow, but return false on error
- returns true on read success


function EOF: boolean;

Returns TRUE if the current position is the end of the input stream


function Next(DataLen: PtrInt): pointer;

Returns the current position, and move ahead the specified bytes


function Next2: cardinal;

Read the next 2 bytes from the buffer as a 16-bit unsigned value


function Next2BigEndian: cardinal;

Read the next 2 bytes from the buffer as a 16-bit big-endian value


function Next4: cardinal;

Read the next 4 bytes from the buffer as a 32-bit unsigned value


function Next8: Qword;

Read the next 8 bytes from the buffer as a 64-bit unsigned value


function NextByte: byte;

Read the next byte from the buffer


function NextByteEquals(Value: byte): boolean;

Consumes the next byte from the buffer, if matches a given value


function NextByteSafe(dest: pointer): boolean;

Read the next byte from the buffer, checking


function NextSafe(out Data: pointer; DataLen: PtrInt): boolean;

Returns the current position, and move ahead the specified bytes


function PeekVarInt32(out value: PtrInt): boolean;

Try to read the next 32-bit signed value from the buffer
- don't change the current position


function PeekVarUInt32(out value: PtrUInt): boolean;

Try to read the next 32-bit unsigned value from the buffer
- don't change the current position


function ReadCompressed(Load: TAlgoCompressLoad = aclNormal; BufferOffset: integer = 0): RawByteString;

Retrieve some TAlgoCompress buffer, appended via Write()
- BufferOffset could be set to reserve some bytes before the uncompressed buffer


function ReadVarRawUtf8DynArray(var Values: TRawUtf8DynArray): PtrInt;

Retrieved RawUtf8 values encoded with TFileBufferWriter.WriteRawUtf8DynArray
- returns the number of items read into Values[] (may differ from length(Values))


function ReadVarUInt32Array(var Values: TIntegerDynArray): PtrInt;

Retrieved cardinal values encoded with TBufferWriter.WriteVarUInt32Array
- Values[] will be resized only if it is not long enough, to spare heap
- returns decoded count in Values[], which may not be length(Values)
- wkFakeMarker will return -count and the caller should make the decoding


function ReadVarUInt64Array(var Values: TInt64DynArray): PtrInt;

Retrieved Int64 values encoded with TBufferWriter.WriteVarUInt64DynArray
- Values[] will be resized only if it is not long enough, to spare heap
- returns decoded count in Values[], which may not be length(Values)


function RemainingLength: PtrUInt;

Returns remaining length (difference between Last and P)


function VarBlob: TValueResult; overload;

Read the next pointer and length value from the buffer


function VarBlobSafe(out Value: TValueResult): boolean;

Read the next pointer and length value from the buffer
- this version won't call ErrorOverflow, but return false on error
- returns true on read success


function VarInt32: integer;

Read the next 32-bit signed value from the buffer


function VarInt64: Int64;

Read the next 64-bit signed value from the buffer


function VarShortString: ShortString;

Read the next ShortString value from the buffer


function VarString: RawByteString; overload;

Read the next RawByteString value from the buffer


function VarString(CodePage: integer): RawByteString; overload;

Read the next RawByteString value from the buffer


function VarUInt32: cardinal;

Read the next 32-bit unsigned value from the buffer


function VarUInt32Safe(out Value: cardinal): boolean;

Read the next 32-bit unsigned value from the buffer
- this version won't call ErrorOverflow, but return false on error
- returns true on read success


function VarUInt64: QWord;

Read the next 64-bit unsigned value from the buffer


function VarUtf8: RawUtf8; overload;

Read the next RawUtf8 value from the buffer


function VarUtf8Safe(out Value: RawUtf8): boolean;

Read the next RawUtf8 value from the buffer
- this version won't call ErrorOverflow, but return false on error
- returns true on read success


procedure Copy(Dest: pointer; DataLen: PtrInt);

Copy data from the current position, and move ahead the specified bytes


procedure ErrorData(const msg: shortstring); overload;

Raise a EFastReader with "Incorrect Data: ...." error message


procedure ErrorData(const fmt: RawUtf8; const args: array of const); overload;

Raise a EFastReader with "Incorrect Data: ...." error message


procedure ErrorOverflow;

Raise a EFastReader with "Reached End of Input" error message


procedure Init(Buffer: pointer; Len: PtrInt); overload;

Initialize the reader from a memory block


procedure Init(const Buffer: RawByteString); overload;

Initialize the reader from a RawByteString content


procedure VarBlob(out result: TValueResult); overload;

Read the next pointer and length value from the buffer


procedure VarBlob(out Value: TSynTempBuffer); overload;

Copy the next VarBlob value from the buffer into a TSynTempBuffer


procedure VarNextInt(count: integer); overload;

Fast ignore the next count VarUInt32/VarInt32/VarUInt64/VarInt64 values
- don't raise any exception, so caller could check explicitly for any EOF


procedure VarNextInt; overload;

Fast ignore the next VarUInt32/VarInt32/VarUInt64/VarInt64 value
- don't raise any exception, so caller could check explicitly for any EOF


procedure VarUtf8(out result: RawUtf8); overload;

Read the next RawUtf8 value from the buffer


1.3.10. EBufferException

EBufferException = class(ESynException)

Exception raised during buffer processing


1.3.11. TBufferWriter

TBufferWriter = class(TObject)

This class can be used to speed up writing to a file or a stream
- big speed up if data is written in small blocks
- also handle optimized storage of any integer/Int64/RawUtf8 values
- use TFileBufferReader or TFastReader for decoding of the stored binary


constructor Create(aClass: TStreamClass; BufLen: integer = 4096); overload;

Initialize the buffer, using an owned TStream instance
- parameter could be e.g. TMemoryStream or TRawByteStringStream
- use Flush then TMemoryStream(Stream) to retrieve its content, or FlushTo if TRawByteStringStream was used
- Write() fails over 800MB (_STRMAXSIZE) for a TRawByteStringStream


constructor Create(aClass: TStreamClass; aTempBuf: pointer; aTempLen: integer); overload;

Initialize with a specified buffer and an owned TStream
- use a specified external buffer (which may be allocated on stack), to avoid a memory allocation
- aClass could be e.g. TMemoryStream or TRawByteStringStream


constructor Create(const aStackBuffer: TTextWriterStackBuffer); overload;

Initialize with a stack-allocated 8KB of buffer
- destination stream is an owned TRawByteStringStream - so you can call FlushTo to retrieve all written data
- Write() fails over 800MB (_STRMAXSIZE) for a TRawByteStringStream
- convenient to reduce heap presure, when writing a few KB of data


constructor Create(aStream: TStream; aTempBuf: pointer; aTempLen: integer); overload;

Initialize with a specified buffer and an existing TStream instance
- use a specified external buffer (which may be allocated on stack), to avoid a memory allocation


constructor Create(aFile: THandle; BufLen: integer = 65536); overload;

Initialize the buffer, and specify a file handle to use for writing
- define an internal buffer of the specified size


constructor Create(aStream: TStream; BufLen: integer = 65536); overload;

Initialize the buffer, and specify a TStream to use for writing
- define an internal buffer of the specified size


constructor Create(const aFileName: TFileName; BufLen: integer = 65536; Append: boolean = false); overload;

Initialize the buffer, and specify a file to use for writing
- define an internal buffer of the specified size
- would replace any existing file by default, unless Append is TRUE


destructor Destroy; override;

Release internal TStream (after AssignToHandle call)
- warning: an explicit call to Flush is needed to write the data pending in internal buffer


function DirectWritePrepare(maxlen: PtrInt; var tmp: RawByteString): PAnsiChar;

Allows to write directly to a memory buffer
- caller should specify the maximum possible number of bytes to be written
- then write the data to the returned pointer, and call DirectWriteFlush
- if len is bigger than the internal buffer, tmp will be used instead


function DirectWriteReserve(maxlen: PtrInt): PByte;

Allows to write directly to a memory buffer
- caller should specify the maximum possible number of bytes to be written
- len should be smaller than the internal buffer size (not checked)


function Flush: Int64;

Write any pending data in the internal buffer to the stream
- after a Flush, it's possible to call FileSeek64(aFile,....)
- returns the number of bytes written between two FLush method calls


function FlushAndCompress(nocompression: boolean = false; algo: TAlgoCompress = nil; BufferOffset: integer = 0): RawByteString;

Write any pending data, then call algo.Compress() on the buffer
- if algo is left to its default nil, will use global AlgoSynLZ
- features direct compression from internal buffer, if stream was not used
- BufferOffset could be set to reserve some bytes before the compressed buffer
- raise an exception if internal Stream is not a TRawByteStringStream


function FlushTo: RawByteString;

Write any pending data, then create a RawByteString from the content
- raise an exception if internal Stream is not a TRawByteStringStream


function FlushToBytes: TBytes;

Write any pending data, then create a TBytes array from the content
- raise an exception if the size exceeds 800MB (_DAMAXSIZE)


procedure CancelAll; virtual;

Rewind the Stream to the position when Create() was called
- note that this does not clear the Stream content itself, just move back its writing position to its initial place


procedure DirectWriteFlush(len: PtrInt; const tmp: RawByteString);

Finalize a direct write to a memory buffer
- by specifying the number of bytes written to the buffer


procedure DirectWriteReserved(pos: PByte);

Flush DirectWriteReserve() content


procedure Write(const Text: RawByteString); overload;

Append some length-prefixed UTF-8 text at the current position
- will write the string length (as VarUInt32), then the string content, as expected by the FromVarString() function
- is just a wrapper around WriteVar()


procedure Write(Data: pointer; DataLen: PtrInt); overload;

Append some data at the current position
- will be inlined as a MoveFast() most of the time


procedure Write1(Data: byte);

Append 1 byte of data at the current position


procedure Write2(Data: cardinal);

Append 2 bytes of data at the current position


procedure Write2BigEndian(Data: cardinal);

Append 2 bytes of data, encoded as BigEndian, at the current position


procedure Write4(Data: integer);

Append 4 bytes of data at the current position


procedure Write4BigEndian(Data: integer);

Append 4 bytes of data, encoded as BigEndian, at the current position


procedure Write8(Data8Bytes: pointer);

Append 8 bytes of data at the current position


procedure WriteBinary(const Data: RawByteString);

Append some content at the current position
- will write the binary data, without any length prefix


procedure WriteI64(Data: Int64);

Append 8 bytes of 64-bit integer at the current position


procedure WriteN(Data: byte; Count: integer);

Append the same byte a given number of occurrences at the current position


procedure WriteRawUtf8Array(Values: PPtrUIntArray; ValuesCount: integer);

Append a RawUtf8 array of values, from its low-level memory pointer
- handled the fixed size strings array case in a very efficient way


procedure WriteRawUtf8DynArray(const Values: TRawUtf8DynArray; ValuesCount: integer);

Append the RawUtf8 dynamic array
- handled the fixed size strings array case in a very efficient way


procedure WriteShort(const Text: ShortString);

Append some UTF-8 encoded text at the current position
- will write the string length (as VarUInt32), then the string content
- is just a wrapper around WriteVar()


procedure WriteStream(aStream: TCustomMemoryStream; aStreamSize: integer = -1);

Append a TStream content
- is StreamSize is left as -1, the Stream.Size is used
- the size of the content is stored in the resulting stream


procedure WriteVar(var Item: TTempUtf8); overload;

Append some TTempUtf8 text content prefixed by its encoded length
- will also release any memory stored in Item.TempRawUtf8


procedure WriteVar(Data: pointer; DataLen: PtrInt); overload;

Append some content (may be text or binary) prefixed by its encoded length
- will write DataLen as VarUInt32, then the Data content, as expected by FromVarString/FromVarBlob functions


procedure WriteVarInt32(Value: PtrInt);

Append an integer value using 32-bit variable-length integer encoding of the by-two complement of the given value


procedure WriteVarInt64(Value: Int64);

Append an integer value using 64-bit variable-length integer encoding of the by-two complement of the given value


procedure WriteVarUInt32(Value: PtrUInt);

Append a cardinal value using 32-bit variable-length integer encoding


procedure WriteVarUInt32Array(const Values: TIntegerDynArray; ValuesCount: integer; DataLayout: TBufferWriterKind);

Append cardinal values (NONE must be negative!) using 32-bit variable-length integer encoding or other specialized algorithm, depending on the data layout
- could be decoded later on via TFastReader.ReadVarUInt32Array


procedure WriteVarUInt32Values(Values: PIntegerArray; ValuesCount: integer; DataLayout: TBufferWriterKind);

Append cardinal values (NONE must be negative!) using 32-bit variable-length integer encoding or other specialized algorithms, depending on the data layout
- could be decoded later on via TFastReader.ReadVarUInt32Array


procedure WriteVarUInt64(Value: QWord);

Append an unsigned integer value using 64-bit variable-length encoding


procedure WriteVarUInt64DynArray(const Values: TInt64DynArray; ValuesCount: integer; Offset: boolean);

Append UInt64 values using 64-bit variable length integer encoding
- if Offset is TRUE, then it will store the difference between two values using 64-bit variable-length integer encoding (in this case, a fixed-sized record storage is also handled separately)
- could be decoded later on via TFastReader.ReadVarUInt64Array


procedure WriteXor(New, Old: PAnsiChar; Len: PtrInt; crc: PCardinal = nil);

Append "New[0..Len-1] xor Old[0..Len-1]" bytes
- as used e.g. by ZeroCompressXor/TSynBloomFilterDiff.SaveTo


property BufferPosition: PtrInt read fPos;

The current position in the internal buffer


property Stream: TStream read fStream;

The associated writing stream


property Tag: PtrInt read fTag write fTag;

Simple property used to store some integer content


property TotalWritten: Int64 read GetTotalWritten;

Get the byte count written since last Flush


1.3.12. TMultiPart

TMultiPart = record

Used by MultiPartFormDataDecode() to return one item of its data


1.3.13. TMemoryMapText

TMemoryMapText = class(TSynPersistent)

Able to read a UTF-8 text file using memory map
- much faster than TStringList.LoadFromFile()
- will ignore any trailing UTF-8 BOM in the file content, but will not expect one either


constructor Create(aFileContent: PUtf8Char; aFileSize: integer); overload;

Read an UTF-8 encoded text file content
- every line beginning is stored into LinePointers[]
- this overloaded constructor accept an existing memory buffer (some uncompressed data e.g.)


constructor Create(const aFileName: TFileName); overload;

Read an UTF-8 encoded text file
- every line beginning is stored into LinePointers[]


destructor Destroy; override;

Release the memory map and internal LinePointers[]


function LineContains(const aUpperSearch: RawUtf8; aIndex: integer): boolean; virtual;

Returns TRUE if the supplied text is contained in the corresponding line


function LineSize(aIndex: integer): integer;

Retrieve the number of UTF-8 chars of the given line
- warning: no range check is performed about supplied index


function LineSizeSmallerThan(aIndex, aMinimalCount: integer): boolean;

Check if there is at least a given number of UTF-8 chars in the given line
- this is faster than LineSize(aIndex)<aMinimalCount for big lines


procedure AddInMemoryLine(const aNewLine: RawUtf8); virtual;

Add a new line to the already parsed content
- this line won't be stored in the memory mapped file, but stay in memory and appended to the existing lines, until this instance is released


procedure AddInMemoryLinesClear; virtual;

Clear all in-memory appended rows


procedure SaveToFile(FileName: TFileName; const Header: RawUtf8 = '');

Save the whole content into a specified file
- including any runtime appended values via AddInMemoryLine()
- an optional header text can be added to the beginning of the file


procedure SaveToStream(Dest: TStream; const Header: RawUtf8);

Save the whole content into a specified stream
- including any runtime appended values via AddInMemoryLine()


property Count: integer read fCount;

The number of text lines


property FileName: TFileName read fFileName write fFileName;

The file name which was opened by this instance


property LinePointers: PPointerArray read fLines;

Direct access to each text line
- use LineSize() method to retrieve line length, since end of line will NOT end with #0, but with #13 or #10
- warning: no range check is performed about supplied index


property Lines[aIndex: integer]: RawUtf8 read GetLine;

Retrieve a line content as UTF-8
- a temporary UTF-8 string is created
- will return '' if aIndex is out of range


property Map: TMemoryMap read fMap;

The memory map used to access the raw file content


property Strings[aIndex: integer]: string read GetString;

Retrieve a line content as RTL string type
- a temporary RTL string is created (after conversion for UNICODE Delphi)
- will return '' if aIndex is out of range


1.3.14. TProgressInfo

TProgressInfo = object(TObject)

Information about the progression of a process, e.g. for TStreamRedirect
- can also compute user-level text information from raw numbers


Context: RawUtf8;

Optional process context, e.g. a download URI, used for logging/progress


CurrentSize: Int64;

Number of bytes for the processed size


Elapsed: Int64;

Number of milliseconds elasped since process beginning


ExpectedSize: Int64;

Number of bytes for the final processed size
- may equal 0 if not known


OnLog: TSynLogProc;

Can be assigned from TSynLog.DoLog class method for low-level logging
- at least at process startup and finish, and every second (ReportDelay)


OnProgress: TOnInfoProgress;

Can be assigned to a TOnInfoProgress callback for high-level logging
- at least at process startup and finish, and every second (ReportDelay)


Percent: integer;

Percentage of CurrentSize versus ExpectedSize
- equals 0 if ExpectedSize is 0


PerSecond: PtrInt;

Number of bytes processed per second


ProcessedSize: Int64;

How many bytes have processed yet


Remaining: Int64;

Number of milliseconds remaining for full process, as estimated
- equals 0 if ExpectedSize is 0
- is just an estimation based on the average PerSecond speed


ReportDelay: Int64;

Number of milliseconds between each DoReport notification
- default is 1000, i.e. to notify OnLog/OnProgress every second


function DoReport(Sender: TObject; ReComputeElapsed: boolean): boolean;

Update the computed fields according to the curent state
- will be updated only every ReportDelay ms (default 1000 = every second)
- return false and compute nothing if ReportDelay has not been reached
- optionally call OnLog and OnProgress callbacks


function GetProgress: RawUtf8;

Retrieve the current status as simple text
- ready to be displayed on the console, e.g. as a single short line with no CRLF during process, and eventually with full information and ending CRLF


procedure DoAfter(Sender: TObject; ChunkSize: Int64);

Can be called


procedure DoStart(Sender: TObject; SizeExpected: Int64; const Ident: string);

Initialize the information for a new process
- once expected size and ident are set, caller should call DoAfter()


procedure Init;

Initialize the information, especially start the timing


procedure SetExpectedSize(SizeExpected, Position: Int64);

Called during process to setup ExpectedSize/ExpectedWrittenSize fields


1.3.15. EStreamRedirect

EStreamRedirect = class(ESynException)

Exception raised during TStreamRedirect processing


1.3.16. TStreamRedirect

TStreamRedirect = class(TStreamWithPosition)

An abstract pipeline stream able to redirect and hash read/written content
- can be used either Read() or Write() calls during its livetime
- hashing is performed on the fly during the Read/Write process
- it features also a callback to mark its progress
- can sleep during Read/Write to reach a LimitPerSecond average bandwidth


constructor Create(aRedirected: TStream; aRead: boolean = false); reintroduce; virtual;

Initialize the internal structure, and start the timing
- before calling Read/Write, you should set the Redirected property or specify aRedirected here - which will be owned by this instance
- if aRead is true, ExpectedSize is set from aRedirected.Size


destructor Destroy; override;

Release the associated Redirected stream


function GetHash: RawUtf8; virtual;

Return the current state of the hash as lower hexadecimal
- by default, will return '' meaning that no hashing algorithm was set


class function GetHashFileExt: RawUtf8; virtual;

Current algorithm name as file/url extension, e.g. '.md5' or '.sha256'
- by default, will return '' meaning that no hashing algorithm was set


class function GetHashName: RawUtf8;

Current algorithm name, from GetHashFileExt, e.g. 'md5' or 'sha256'


class function HashFile(const FileName: TFileName; const OnProgress: TOnStreamProgress = nil): RawUtf8;

Apply the internal hash algorithm to the supplied file content
- could be used ahead of time to validate a cached file


function Read(var Buffer; Count: Longint): Longint; override;

Update the hash and redirect the data to the associated TStream
- also trigger OnProgress at least every second
- will raise an error if Write() (or Append) have been called before


function Seek(const Offset: Int64; Origin: TSeekOrigin): Int64; override;

Overriden to support Seek(0, soBeginning) and reset the Redirected stream
- mandatory for proper THttpClientSocket.SockSendStream rewind


function Write(const Buffer; Count: Longint): Longint; override;

Update the hash and redirect the data to the associated TStream
- also trigger OnProgress at least every second
- will raise an error if Read() has been called before


procedure Append;

Update the hash of the existing Redirected stream content
- ready to Write() some new data after the existing


procedure Ended;

Notify end of process
- should be called explicitly when all Read()/Write() has been done


class procedure NotifyEnded( const OnStream: TOnStreamProgress; const OnInfo: TOnInfoProgress; const Fmt: RawUtf8; const Args: array of const; Size, StartedMs: Int64);

Notify a TOnStreamProgress callback that a process ended
- create a fake TStreamRedirect and call Ended with the supplied info


class procedure ProgressInfoToConsole(Sender: TObject; Info: PProgressInfo);

Can be used as TOnInfoProgress callback writing into the console


class procedure ProgressStreamToConsole(Sender: TStreamRedirect);

Can be used as TOnStreamProgress callback writing into the console


procedure Terminate;

Could be set from another thread to abort the streaming process
- will raise an exception at the next Read()/Write() call


property Context: RawUtf8 read fInfo.Context write fInfo.Context;

Optional process context, e.g. a download URI, used for logging/progress


property Elapsed: Int64 read fInfo.Elapsed;

Number of milliseconds elasped since beginning, as set by Read/Write


property ExpectedSize: Int64 read fInfo.ExpectedSize write SetExpectedSize;

You can specify a number of bytes for the final Redirected size
- will be used for the callback progress - could be left to 0 for Write() if size is unknown


property LimitPerSecond: PtrInt read fLimitPerSecond write fLimitPerSecond;

Can limit the Read/Write bytes-per-second bandwidth used, if not 0
- sleep so that PerSecond will keep close to this LimitPerSecond value


property OnInfoProgress: TOnInfoProgress read fInfo.OnProgress write fInfo.OnProgress;

Optional TOnInfoProgress callback triggered during Read/Write
- at least at process startup and finish, and every second / ReportDelay


property OnLog: TSynLogProc read fInfo.OnLog write fInfo.OnLog;

Can be assigned from TSynLog.DoLog class method for low-level logging


property OnProgress: TOnStreamProgress read fOnStreamProgress write fOnStreamProgress;

Optional TOnStreamProgress callback triggered during Read/Write
- at least at process startup and finish, and every second / ReportDelay


property Percent: integer read fInfo.Percent;

Percentage of Size versus ExpectedSize
- equals 0 if ExpectedSize is 0


property PerSecond: PtrInt read fInfo.PerSecond;

Number of bytes processed per second, since initialization of this instance


property ProcessedSize: Int64 read fInfo.ProcessedSize;

How many bytes have passed through Read() or Write()
- may not equal Size or Position after an Append - e.g. on resumed download from partial file


property Progress: RawUtf8 read GetProgress;

The current progression as text, as returned by ProgressStreamToConsole


property Redirected: TStream read fRedirected write fRedirected;

Specify a TStream to which any Read()/Write() will be redirected
- this TStream instance will be owned by the TStreamRedirect


property Remaining: Int64 read fInfo.Remaining;

Number of milliseconds remaining for full process, as set by Read/Write
- equals 0 if ExpectedSize is 0
- is just an estimation based on the average PerSecond speed


property ReportDelay: Int64 read fInfo.ReportDelay write fInfo.ReportDelay;

Number of milliseconds between each notification
- default is 1000, i.e. notify OnLog/OnProgress/OnInfoProgress every second


property TimeOut: Int64 read fTimeOut write fTimeOut;

Read/Write will raise an exception if not finished after TimeOut milliseconds


1.3.17. TStreamRedirectHash32

TStreamRedirectHash32 = class(TStreamRedirect)

TStreamRedirect with 32-bit THasher checksum


1.3.18. TStreamRedirectCrc32c

TStreamRedirectCrc32c = class(TStreamRedirectHash32)

TStreamRedirect with crc32c 32-bit checksum


1.3.19. TFakeWriterStream

TFakeWriterStream = class(TStream)

A fake TStream, which will just count the number of bytes written


1.3.20. TNestedStreamReader

TNestedStreamReader = class(TStreamWithPositionAndSize)

TStream allowing to read from some nested TStream instances


destructor Destroy; override;

Finalize the nested TStream instance


function ForText: TRawByteStringStream;

Get the last TRawByteStringStream, or append a new one if needed


function NewStream(Stream: TStream): TStream;

Append a nested TStream instance
- you could use a TFileStreamEx here for efficient chunked reading


function Read(var Buffer; Count: Longint): Longint; override;

Will read up to Count bytes from the internal nested TStream


function Seek(const Offset: Int64; Origin: TSeekOrigin): Int64; override;

Overriden method to call Flush on rewind, i.e. if position is set to 0


function Write(const Buffer; Count: Longint): Longint; override;

This TStream is read-only: calling this method will raise an exception


procedure Append(const Content: RawByteString);

Append some text or content to an internal TRawByteStringStream
- is the easy way to append some text or data to the internal buffers


procedure Flush; virtual;

You should call this method before any Read() call
- is also called when you execute Seek(0, soBeginning)


1.3.21. TBufferedStreamReader

TBufferedStreamReader = class(TStreamWithPositionAndSize)

TStream with an internal memory buffer
- can be beneficial e.g. reading from a file by small chunks


constructor Create(const aSourceFileName: TFileName; aBufSize: integer = 65536); reintroduce; overload;

Initialize a source file and the internal buffer


constructor Create(aSource: TStream; aBufSize: integer = 65536); reintroduce; overload;

Initialize the source TStream and the internal buffer
- will also rewind the aSource position to its beginning, and retrieve its size


destructor Destroy; override;

Finalize this instance and its buffer


function Read(var Buffer; Count: Longint): Longint; override;

Will read up to Count bytes from the internal buffer or source TStream


function Seek(const Offset: Int64; Origin: TSeekOrigin): Int64; override;

Overriden method to flush buffer on rewind


function Write(const Buffer; Count: Longint): Longint; override;

This TStream is read-only: calling this method will raise an exception


1.3.22. TRawByteStringGroupValue

TRawByteStringGroupValue = record

Item as stored in a TRawByteStringGroup instance


1.3.23. TRawByteStringGroup

TRawByteStringGroup = object(TObject)

Store several RawByteString content with optional concatenation


Count: integer;

How many items are currently stored in Values[]


LastFind: integer;

Naive but efficient cache for Find()


Position: integer;

The current size of data stored in Values[]


Values: TRawByteStringGroupValueDynArray;

Actual list storing the data


function AsBytes: TByteDynArray;

Return all content as a single TByteDynArray


function AsText: RawByteString;

Return all content as a single RawByteString
- will also compact the Values[] array into a single item (which is returned)


function Equals(const aAnother: TRawByteStringGroup): boolean;

Compare two TRawByteStringGroup instance stored text


function Find(aPosition: integer): PRawByteStringGroupValue; overload;

Returns a pointer to Values[] containing a given position
- returns nil if not found


function Find(aPosition, aLength: integer): pointer; overload;

Returns a pointer to Values[].Value containing a given position and length
- returns nil if not found


function FindAsText(aPosition, aLength: integer): RawByteString; overload;

Returns the text at a given position in Values[]
- text should be in a single Values[] entry


procedure Add(const aItem: RawByteString); overload;

Add a new item to Values[]


procedure Add(const aAnother: TRawByteStringGroup); overload;

Add another TRawByteStringGroup to Values[]


procedure Add(aItem: pointer; aItemLen: integer); overload;

Add a new item to Values[]


procedure AddFromReader(var aReader: TFastReader);

Add another TRawByteStringGroup previously serialized via WriteString()


procedure AppendTextAndClear(var aDest: RawByteString);

Append stored information into another RawByteString, and clear content


procedure Clear;

Clear any stored information


procedure Compact;

Compact the Values[] array into a single item
- is also used by AsText to compute a single RawByteString


procedure FindAsText(aPosition, aLength: integer; out aText: RawByteString); overload;

Returns the text at a given position in Values[]
- text should be in a single Values[] entry


procedure FindAsVariant(aPosition, aLength: integer; out aDest: variant);

Returns the text at a given position in Values[] via RawUtf8ToVariant()
- text should be in a single Values[] entry
- explicitly returns null if the supplied text was not found


procedure FindMove(aPosition, aLength: integer; aDest: pointer);

Copy the text at a given position in Values[]
- text should be in a single Values[] entry


procedure FindWrite(aPosition, aLength: integer; W: TTextWriter; Escape: TTextWriterKind = twJsonEscape; TrailingCharsToIgnore: integer = 0);

Append the text at a given position in Values[], JSON escaped by default
- text should be in a single Values[] entry


procedure FindWriteBase64(aPosition, aLength: integer; W: TTextWriter; withMagic: boolean);

Append the blob at a given position in Values[], Base64 encoded
- text should be in a single Values[] entry


procedure RemoveLastAdd;

Low-level method to abort the latest Add() call
- warning: will work only once, if an Add() has actually been just called: otherwise, the behavior is unexpected, and may wrongly truncate data


procedure Write(W: TTextWriter; Escape: TTextWriterKind = twJsonEscape); overload;

Save all content into a TJsonWriter instance


procedure WriteBinary(W: TBufferWriter); overload;

Save all content into a TBufferWriter instance


procedure WriteString(W: TBufferWriter);

Save all content as a string into a TBufferWriter instance
- storing the length as WriteVarUInt32() prefix


1.3.24. TRawByteStringHeap

TRawByteStringHeap = class(TObject)

Thread-safe reusable set of constant RawByteString/RawUtf8 instances
- all RawByteString will be constant (RefCnt=-2) with some max length
- maintain internally its own TLockedList O(1) set of instances
- warning: any call to New() should manually be followed by one Release()


constructor Create(aMaxLength: integer);

Initialize the internal cache for a given maximum length


destructor Destroy; override;

Release all cached instances


function Clean: PtrInt;

Release the RawByteString instances in the cache bin
- keep any existing New() instances intact
- returns how many memory has been released to the heap


procedure New(var aDest: RawByteString; aText: PUtf8Char = nil; aTextLen: TStrLen = 0; aCodePage: integer = CP_RAWBYTESTRING); overload;

Return a new RawByteString of a given length, with refcount=-2
- if aTextLen is its default 0, MaxLength will be used
- returned from its internal cache, unless aTextLen>MaxLength, which will allocate a regular RawByteString from heap


procedure New(var aDest: RawUtf8; aText: PUtf8Char; aTextLen: TStrLen); overload;

Return a new RawUtf8 of a given length, with refcount = -2


procedure Release(var aDest: pointer); overload;

Put back a RawByteString acquired from NewUtf8() into the internal cache


procedure Release(var aDest: RawByteString); overload;

Put back a RawByteString acquired from New() into the internal cache


procedure Release(var aDest: RawUtf8); overload;

Put back a RawUtf8 acquired from New() into the internal cache


property Count: integer read fOne.Count;

How many New() calls are currently active


property MaxLength: TStrLen read fMaxLength;

The maximum length() of RawByteString returned by New()


1.3.25. TRawByteStringBuffer

TRawByteStringBuffer = object(TObject)

Store one RawByteString content with an associated length
- to be used e.g. as a convenient reusable memory buffer


function Buffer: pointer;

A convenient wrapper to pointer(fBuffer) for direct Buffer/Len use


function Capacity: PtrInt;

How many bytes are currently allocated in the Buffer


function Extract(Dest: pointer; Count: PtrInt): PtrInt;

Move up to Count bytes from the internal Buffer into another place
- returns how many bytes were available to be copied into Dest^
- then remove the copied bytes from the internal Buffer/Len storage


function ExtractAt(var Dest: PAnsiChar; var Count: PtrInt; var Pos: PtrInt): PtrInt;

Move up to Count bytes from the internal Buffer into another place
- returns how many bytes were available to be copied into Dest^
- don't move any byte, but just update the given Pos index


function TryAppend(P: pointer; PLen: PtrInt): boolean;

Add some UTF-8 buffer content to the Buffer, without resizing it


procedure Append(const Text: array of RawUtf8); overload;

Add some UTF-8 string(s) content to the Buffer, resizing it if needed


procedure Append(Ch: AnsiChar); overload;

Just after Append/AppendShort, append one single character


procedure Append(const Text: RawUtf8); overload;

Add some UTF-8 string content to the Buffer, resizing it if needed


procedure Append(P: pointer; PLen: PtrInt); overload;

Add some UTF-8 buffer content to the Buffer, resizing it if needed


procedure Append(Value: QWord); overload;

Add some number as text content to the Buffer, resizing it if needed


procedure AppendCRLF;

Just after Append/AppendShort, append a #13#10 end of line


procedure AppendShort(const Text: ShortString);

Add some UTF-8 shortstring content to the Buffer, resizing it if needed


procedure AsText(out Text: RawUtf8; Overhead: PtrInt = 0; UseMainBuffer: boolean = false);

Retrieve the current Buffer/Len content as RawUtf8 text
- with some optional overhead for faster reallocmem at concatenation
- won't force Len to 0: caller should call Reset if done with it
- UseMainBuffer=true will return a copy of fBuffer into Text


procedure Clear;

Release/free the internal Buffer storage


procedure Remove(FirstBytes: PtrInt);

Similar to delete(fBuffer, 1, FirstBytes)


procedure Reserve(MaxSize: PtrInt); overload;

Ensure the internal Buffer has at least MaxSize bytes
- also reset the internal Len to 0


procedure Reserve(const WorkingBuffer: RawByteString); overload;

Use a specified string buffer as start


procedure Reset;

Set Len to 0, but doesn't clear/free the Buffer itself


property Len: PtrInt read fLen write fLen;

How many bytes are currently used in the Buffer


1.4. Types implemented in the mormot.core.buffers unit

1.4.1. PRawByteStringBuffer

PRawByteStringBuffer = ^TRawByteStringBuffer;

Pointer reference to a TRawByteStringBuffer


1.4.2. PRawByteStringGroup

PRawByteStringGroup = ^TRawByteStringGroup;

Pointer reference to a TRawByteStringGroup


1.4.3. TAlgoCompressLoad

TAlgoCompressLoad = ( aclNormal, aclSafeSlow, aclNoCrcFast );

Define the implementation used by TAlgoCompress.Decompress()


1.4.4. TBufferWriterKind

TBufferWriterKind = ( wkUInt32, wkVarUInt32, wkVarInt32, wkSorted, wkOffsetU, wkOffsetI, wkFakeMarker );

Available kind of integer array storage, corresponding to the data layout of TBufferWriter
- wkUInt32 will write the content as "plain" 4 bytes binary (this is the prefered way if the integers can be negative)
- wkVarUInt32 will write the content using our 32-bit variable-length integer encoding
- wkVarInt32 will write the content using our 32-bit variable-length integer encoding and the by-two complement (0=0,1=1,2=-1,3=2,4=-2...)
- wkSorted will write an increasing array of integers, handling the special case of a difference of similar value (e.g. 1) between two values - note that this encoding is efficient only if the difference is mainly < 253
- wkOffsetU and wkOffsetI will write the difference between two successive values, with detection of any constant difference (unsigned or signed)
- wkFakeMarker won't be used by WriteVarUInt32Array, but to notify a custom encoding


1.4.5. TEmoji

TEmoji = ( eNone, eGrinning, eGrin, eJoy, eSmiley, eSmile, eSweat_smile, eLaughing, eInnocent, eSmiling_imp, eWink, eBlush, eYum, eRelieved, eHeart_eyes, eSunglasses, eSmirk, eNeutral_face, eExpressionless, eUnamused, eSweat, ePensive, eConfused, eConfounded, eKissing, eKissing_heart, eKissing_smiling_eyes, eKissing_closed_eyes, eStuck_out_tongue, eStuck_out_tongue_winking_eye, eStuck_out_tongue_closed_eyes, eDisappointed, eWorried, eAngry, ePout, eCry, ePersevere, eTriumph, eDisappointed_relieved, eFrowning, eAnguished, eFearful, eWeary, eSleepy, eTired_face, eGrimacing, eSob, eOpen_mouth, eHushed, eCold_sweat, eScream, eAstonished, eFlushed, eSleeping, eDizzy_face, eNo_mouth, eMask, eSmile_cat, eJoy_cat, eSmiley_cat, eHeart_eyes_cat, eSmirk_cat, eKissing_cat, ePouting_cat, eCrying_cat_face, eScream_cat, eSlightly_frowning_face, eSlightly_smiling_face, eUpside_down_face, eRoll_eyes, eNo_good, oOk_woman, eBow, eSee_no_evil, eHear_no_evil, eSpeak_no_evil, eRaising_hand, eRaised_hands, eFrowning_woman, ePerson_with_pouting_face, ePray );

Map the first Unicode page of Emojis, from U+1F600 to U+1F64F
- naming comes from github/Markdown :identifiers:


1.4.6. TFileBufferWriter

TFileBufferWriter = TBufferWriter;

Deprecated alias to TBufferWriter binary serializer


1.4.7. THashFile

THashFile = function(const FileName: TFileName): RawUtf8;

Prototype of a file hashing function, returning its hexadecimal hash
- match HashFileCrc32c() below, HashFileCrc32() in mormot.core.zip, and HashFileMd5/HashFileSha* in mormot.crypt.secure functions signature


1.4.8. TLogEscape

TLogEscape = array[0..511] of AnsiChar;

512 bytes buffer to be allocated on stack when using LogEscape()


1.4.9. TMimeType

TMimeType = ( mtUnknown, mtPng, mtGif, mtTiff, mtJpg, mtBmp, mtDoc, mtPpt, mtXls, mtHtml, mtCss, mtJS, mtXIcon, mtFont, mtText, mtSvg, mtXml, mtWebp, mtManifest, mtJson, mtOgg, mtMp4, mtMp2, mtMpeg, mtH264, mtWma, mtWmv, mtAvi, mtGzip, mtWebm, mtRar, mt7z, mtZip, mtBz2, mtPdf, mtSQlite3, mtXcomp, mtDicom );

Some of the best-known mime types
- subset of the whole IANA list which can be quite huge (>1500 items)


1.4.10. TMultiPartDynArray

TMultiPartDynArray = array of TMultiPart;

Used by MultiPartFormDataDecode() to return all its data items


1.4.11. TOnFastReaderErrorData

TOnFastReaderErrorData = procedure(const fmt: RawUtf8; const args: array of const) of object;

Event signature to customize TFastReader.ErrorData notification


1.4.12. TOnFastReaderErrorOverflow

TOnFastReaderErrorOverflow = procedure of object;

Event signature to customize TFastReader.ErrorOverflow notification


1.4.13. TOnInfoProgress

TOnInfoProgress = procedure(Sender: TObject; Info: PProgressInfo) of object;

Callback definition to notify some TProgressInfo process
- see e.g. TStreamRedirect.ProgressInfoToConsole


1.4.14. TOnStreamCreate

TOnStreamCreate = function(const FileName: string; Mode: cardinal): TStream of object;

Optional callback as used e.g. by THttpClientSocketWGet.OnStreamCreate


1.4.15. TOnStreamProgress

TOnStreamProgress = procedure(Sender: TStreamRedirect) of object;

TStreamHasher.Write optional progression callback
- see Sender properties like Context/Size/PerSecond and ExpectedSize (which may be 0 if the download size is unknown)
- see e.g. TStreamRedirect.ProgressStreamToConsole


1.4.16. TRawByteStringGroupValueDynArray

TRawByteStringGroupValueDynArray = array of TRawByteStringGroupValue;

Items as stored in a TRawByteStringGroup instance


1.4.17. TStreamRedirectClass

TStreamRedirectClass = class of TStreamRedirect;

Meta-class of TStreamRedirect hierarchy


1.4.18. TTextWriterHtmlEscape

TTextWriterHtmlEscape = set of ( heHtmlEscape, heEmojiToUtf8);

Tune AddHtmlEscapeWiki/AddHtmlEscapeMarkdown wrapper functions process
- heHtmlEscape will escape any HTML special chars, e.g. & into &amp;
- heEmojiToUtf8 will convert any Emoji text into UTF-8 Unicode character, recognizing e.g. :joy: or :) in the text


1.4.19. TUrlEncoder

TUrlEncoder = set of ( ueTrimLeadingQuestionMark, ueEncodeNames, ueStarNameIsCsv, ueSkipVoidString, ueSkipVoidValue);

Some options for UrlEncode()


1.5. Constants implemented in the mormot.core.buffers unit

1.5.1. ALGO_SAFE

ALGO_SAFE: array[boolean] of TAlgoCompressLoad = ( aclNormal, aclSafeSlow);

Used e.g. as when ALGO_SAFE[SafeDecompression] for TAlgoCompress.Decompress


1.5.2. JSON_BASE64_MAGIC_C

JSON_BASE64_MAGIC_C = $b0bfef;

UTF-8 encoded \uFFF0 special code to mark Base64 binary content in JSON
- Unicode special char U+FFF0 is UTF-8 encoded as EF BF B0 bytes
- as generated by BinToBase64WithMagic() functions, and expected by the TExtractInlineParameters decoder
- used e.g. when transmitting TDynArray.SaveTo() content


1.5.3. JSON_BASE64_MAGIC_QUOTE_C

JSON_BASE64_MAGIC_QUOTE_C = ord('"') + cardinal(JSON_BASE64_MAGIC_C) shl 8;

'"' + UTF-8 encoded \uFFF0 special code to mark Base64 binary in JSON


1.5.4. JSON_BASE64_MAGIC_QUOTE_S

JSON_BASE64_MAGIC_QUOTE_S: string[4] = '"'#$ef#$bf#$b0;

'"' + UTF-8 encoded \uFFF0 special code to mark Base64 binary in JSON
- defined as a ShortString constant to be used as:

 AddShorter(JSON_BASE64_MAGIC_QUOTE_S);

1.5.5. JSON_BASE64_MAGIC_S

JSON_BASE64_MAGIC_S: string[3] = #$ef#$bf#$b0;

UTF-8 encoded \uFFF0 special code to mark Base64 binary content in JSON
- Unicode special char U+FFF0 is UTF-8 encoded as EF BF B0 bytes
- as generated by BinToBase64WithMagic() functions, and expected by the TExtractInlineParameters decoder
- used e.g. when transmitting TDynArray.SaveTo() content


1.5.6. LOG_MAGIC

LOG_MAGIC = $ABA51051;

The "magic" number used to identify .log.synlz compressed files, as created by EventArchiveSynLZ / EventArchiveLizard callbacks


1.5.7. MIME_TYPE

MIME_TYPE: array[TMimeType] of RawUtf8 = ( '', 'image/png', 'image/gif', 'image/tiff', JPEG_CONTENT_TYPE, 'image/bmp', 'application/msword', 'application/vnd.ms-powerpoint', 'application/vnd.ms-excel', HTML_CONTENT_TYPE, 'text/css', 'text/javascript', 'image/x-icon', 'font/woff', TEXT_CONTENT_TYPE, 'image/svg+xml', XML_CONTENT_TYPE, 'image/webp', 'text/cache-manifest', JSON_CONTENT_TYPE, 'video/ogg', 'video/mp4', 'video/mp2', 'audio/mpeg', 'video/H264', 'audio/x-ms-wma', 'video/x-ms-wmv', 'video/x-msvideo', 'application/gzip', 'video/webm', 'application/x-rar-compressed', 'application/x-7z-compressed', 'application/zip', 'application/bzip2', 'application/pdf', 'application/x-sqlite3', 'application/x-compress', 'application/dicom');

The known mime types text representation


1.5.8. PLURAL_FORM

PLURAL_FORM: array[boolean] of RawUtf8 = ( '', 's');

Can be used to append to most English nouns to form a plural
- as used by the Plural() function


1.5.9. SYNLZTRIG

SYNLZTRIG: array[boolean] of integer = ( 100, maxInt);

CompressionSizeTrigger parameter SYNLZTRIG[true] will disable then SynLZCompress() compression


1.6. Functions or procedures implemented in the mormot.core.buffers unit

Functions or proceduresDescription
AddHtmlEscapeMarkdownConvert minimal Markdown text into proper HTML
AddHtmlEscapeWikiConvert some wiki-like text into proper HTML
Append999ToBufferFast add text conversion of 0-999 integer value into a given buffer
AppendBuffersToRawUtf8Fast add some characters to a RawUtf8 string
AppendBufferToBufferFast add some characters from ane buffer into another buffer
AppendBufferToRawByteStringJust redirect to mormot.core.text Append(...) overloads
AppendCharOnceToRawUtf8Fast add one character to a RawUtf8 string, if not already present
AppendRawUtf8ToBufferFast add some characters from a RawUtf8 string into a given buffer
AppendUInt32ToBufferFast add text conversion of a 32-bit signed integer value into a given buffer
AreUrlValidChecks if the supplied UTF-8 text values don't need URI encoding
AsciiToBaudotConvert some ASCII-7 text into binary, using Emile Baudot code
AsciiToBaudotConvert some ASCII-7 text into binary, using Emile Baudot code
Base32ToBinConversion from Base32 encoded text into a binary string
Base32ToBinConversion from Base32 encoded text into a binary string
Base58ToBinConversion from Base58 encoded text into a binary string
Base58ToBinConversion from Base58 encoded text into a binary string
Base58ToBinConversion from Base58 encoded text into a binary buffer
Base64DecodeDirect low-level decoding of a Base64 encoded buffer
Base64EncodeRaw function for efficient binary to Base64 encoding
Base64EncodeTrailingRaw function for efficient binary to Base64 encoding of the last bytes
Base64MagicCheckAndDecodeCheck and decode '\uFFF0base64encodedbinary' content into binary
Base64MagicCheckAndDecodeCheck and decode '\uFFF0base64encodedbinary' content into binary
Base64MagicCheckAndDecodeCheck and decode '\uFFF0base64encodedbinary' content into binary
Base64MagicDecodeJust a wrapper around Base64ToBin() for in-place decode of JSON_BASE64_MAGIC_C '\uFFF0base64encodedbinary' content into binary
Base64MagicToBlobConvert a Base64-encoded content into binary hexadecimal ready for SQL
Base64MagicTryAndDecodeDecode '\uFFF0base64encodedbinary' or 'base64encodedbinary' into binary
Base64ToBinFast conversion from Base64 encoded text into binary data
Base64ToBinFast conversion from Base64 encoded text into binary data
Base64ToBinFast conversion from Base64 encoded text into binary data
Base64ToBinFast conversion from Base64 encoded text into binary data
Base64ToBinFast conversion from Base64 encoded text into binary data
Base64ToBinFast conversion from Base64 encoded text into binary data
Base64ToBinLengthRetrieve the expected undecoded length of a Base64 encoded buffer
Base64ToBinLengthSafeRetrieve the expected undecoded length of a Base64 encoded buffer
Base64ToBinSafeFast conversion from Base64 encoded text into binary data
Base64ToBinSafeFast conversion from Base64 encoded text into binary data
Base64ToBinSafeFast conversion from Base64 encoded text into binary data
Base64ToBinSafeFast conversion from Base64 encoded text into binary data
Base64ToBinTrimConversion from Base64 encoded text into binary data, ignoring spaces
Base64ToUriConversion from any Base64 encoded value into URI-compatible encoded text
Base64uriDecodeDirect low-level decoding of a Base64-URI encoded buffer
Base64uriEncodeLow-level conversion from a binary buffer into Base64-like URI-compatible encoded text
Base64uriToBinFast conversion from Base64-URI encoded text into binary data
Base64uriToBinFast conversion from Base64-URI encoded text into binary data
Base64uriToBinFast conversion from Base64-URI encoded text into binary data
Base64uriToBinFast conversion from Base64-URI encoded text into binary data
Base64uriToBinFast conversion from Base64-URI encoded text into binary data
Base64uriToBinFast conversion from Base64-URI encoded text into binary data
Base64uriToBinLengthRetrieve the expected undecoded length of a Base64-URI encoded buffer
BaudotToAsciiConvert some Baudot code binary, into ASCII-7 text
BaudotToAsciiConvert some Baudot code binary, into ASCII-7 text
BinToBase32Conversion from a binary buffer into Base32 encoded text as RawUtf8
BinToBase32Conversion from a binary buffer into Base32 encoded text as RawUtf8
BinToBase32Conversion from a binary buffer into Base32 encoded text buffer
BinToBase32LengthCompute the length resulting of Base32 encoding of a binary buffer
BinToBase58Conversion from a binary buffer into Base58 encoded text as TSynTempBuffer
BinToBase58Conversion from a binary buffer into Base58 encoded text as RawUtf8
BinToBase58Conversion from a binary buffer into Base58 encoded text as RawUtf8
BinToBase64Fast conversion from binary data into prefixed/suffixed Base64 encoded UTF-8 text
BinToBase64Fast conversion from binary data into Base64 encoded UTF-8 text
BinToBase64Fast conversion from binary data into Base64 encoded UTF-8 text
BinToBase64LengthRetrieve the expected encoded length after Base64 process
BinToBase64LineFast conversion from binary into prefixed/suffixed Base64 with 64 chars per line
BinToBase64ShortFast conversion from a small binary data into Base64 encoded UTF-8 text
BinToBase64ShortFast conversion from a small binary data into Base64 encoded UTF-8 text
BinToBase64uriFast conversion from binary data into Base64-like URI-compatible encoded text
BinToBase64uriFast conversion from a binary buffer into Base64-like URI-compatible encoded text
BinToBase64uriLengthRetrieve the expected encoded length after Base64-URI process
BinToBase64uriShortFast conversion from a binary buffer into Base64-like URI-compatible encoded ShortString
BinToBase64WithMagicFast conversion from binary data into Base64 encoded UTF-8 text with JSON_BASE64_MAGIC_C prefix (UTF-8 encoded \uFFF0 special code)
BinToBase64WithMagicFast conversion from binary data into Base64 encoded UTF-8 text with JSON_BASE64_MAGIC_C prefix (UTF-8 encoded \uFFF0 special code)
BinToBase64WithMagicFast conversion from binary data into Base64 encoded UTF-8 text with JSON_BASE64_MAGIC_C prefix (UTF-8 encoded \uFFF0 special code)
BinToHumanHexGenerate some 'xx:xx:xx:xx' output buffer with left and right margins
BinToHumanHexGenerate some 'xx:xx:xx:xx' output buffer with left and right margins
BinToSourceGenerate some pascal source code holding some data binary as constant
BinToSourceGenerate some pascal source code holding some data binary as constant
BinToSourceGenerate some pascal source code holding some data binary as constant
BlobToBytesCreate a TBytes from TEXT-encoded blob data
BlobToRawBlobFill a RawBlob from TEXT-encoded blob data
BlobToRawBlobFill a RawBlob from TEXT-encoded blob data
BlobToRawBlobFill a RawBlob from TEXT-encoded blob data
BlobToStreamCreate a memory stream from TEXT-encoded blob data
BytesToRawByteStringCreates a RawByteString memory buffer from a TBytes content
ContentToShortIf source is not UTF-8 calls EscapeToShort, otherwise return it directly
EmojiFromDotsLow-level conversion of github/Markdown :identifiers: into UTF-8 Emoji sequences
EmojiFromDotsConversion of github/Markdown :identifiers: into UTF-8 Emoji sequences
EmojiFromTextRecognize github/Markdown compatible text of Emojis
EmojiParseDotsLow-level parser of github/Markdown compatible text of Emojis
EmojiToDotsLow-level conversion of UTF-8 Emoji sequences into github/Markdown :identifiers:
EmojiToDotsConversion of UTF-8 Emoji sequences into github/Markdown :identifiers:
EscapeBufferLow-level fast conversion from binary data to escaped text
EscapeToShortFill a ShortString with the (hexadecimal) chars of the input text/binary
EscapeToShortFill a ShortString with the (hexadecimal) chars of the input text/binary
FromVarBlobRetrieve pointer and length to a variable-length text/blob buffer
FromVarInt32Convert a 32-bit variable-length integer buffer into an integer
FromVarInt64Convert a 64-bit variable-length integer buffer into a Int64
FromVarInt64ValueConvert a 64-bit variable-length integer buffer into a Int64
FromVarStringRetrieve a variable-length UTF-8 encoded text buffer in a newly allocation RawUtf8
FromVarStringRetrieve a variable-length UTF-8 encoded text buffer in a temporary buffer
FromVarStringRetrieve a variable-length UTF-8 encoded text buffer in a temporary buffer
FromVarStringSafe retrieve a variable-length UTF-8 encoded text buffer in a newly allocation RawUtf8
FromVarStringRetrieve a variable-length text buffer
FromVarStringRetrieve a variable-length text buffer
FromVarStringRetrieve a variable-length UTF-8 encoded text buffer in a newly allocation RawUtf8
FromVarUInt32Convert a 32-bit variable-length integer buffer into a cardinal
FromVarUInt32Convert a 32-bit variable-length integer buffer into a cardinal
FromVarUInt32BigConvert a 32-bit variable-length integer buffer into a cardinal
FromVarUInt32HighConvert a 32-bit variable-length integer buffer into a cardinal
FromVarUInt32SafeSafely convert a 32-bit variable-length integer buffer into a cardinal
FromVarUInt32Up128Convert a 32-bit variable-length integer buffer into a cardinal
FromVarUInt64Convert a 64-bit variable-length integer buffer into a UInt64
FromVarUInt64Convert a 64-bit variable-length integer buffer into a UInt64
FromVarUInt64SafeSafely convert a 64-bit variable-length integer buffer into a UInt64
GetJpegSizeFast guess of the size, in pixels, of a JPEG memory buffer
GetMimeContentTypeRetrieve the MIME content type from its file name or a supplied binary buffer
GetMimeContentTypeFromBufferRetrieve the MIME content type from a supplied binary buffer
GetMimeContentTypeFromExtMtUnknown mtPng mtGif mtTiff mtJpg mtBmp mtDoc mtPpt mtXls mtHtml mtCss mtJS RFC 9239 mtXIcon mtFont RFC 8081 mtText mtSvg mtXml mtWebp mtManifest mtJson mtOgg RFC 5334 mtMp4 RFC 4337 6381 mtMp2 mtMpeg RFC 3003 mtH264 RFC 6184 mtWma mtWmv mtAvi mtGzip mtWebm mtRar mt7z mtZip mtBz2 mtPdf mtSQlite3 mtXcomp mtDicom retrieve the MIME content type from its file name
GetMimeContentTypeFromMemoryRetrieve the MIME content type from a supplied binary buffer
GetMimeContentTypeHeaderRetrieve the HTTP header for MIME content type from a supplied binary buffer
GetMimeTypeFromExtRetrieve the MIME content type from its file extension text (without '.')
GetStreamBufferRetrieve the memory buffer of a TCustomMemoryStream/TRawByteStringStream
GotoNextVarIntJump a value in the 32-bit or 64-bit variable-length integer buffer
GotoNextVarStringJump a value in variable-length text buffer
HashFileCompute the 32-bit default hash of a file content
HashFileCrc32cCompute the crc32c checksum of a given file
HtmlEscapeMarkdownEscape some Markdown-marked text into HTML
HtmlEscapeWikiEscape some wiki-marked text into HTML
IncludeTrailingUriDelimiterEnsure the supplied URI contains a trailing '/' charater
IsBase64Check if the supplied text is a valid Base64 encoded stream
IsBase64Check if the supplied text is a valid Base64 encoded stream
isBlobHexReturn true if the TEXT is encoded as SQLite3 BLOB literals (X'53514C697465' e.g.)
IsContentCompressedRetrieve if some content is compressed, from a supplied binary buffer
IsContentTypeCompressibleRecognize e.g. 'text/css' or 'application/json' as compressible
IsContentTypeCompressibleURecognize e.g. 'text/css' or 'application/json' as compressible
IsContentTypeJsonRecognize e.g. 'application/json' or 'application/vnd.api+json'
IsStreamBufferCheck if class is a TCustomMemoryStream/TRawByteStringStream
IsUrlValidChecks if the supplied UTF-8 text don't need URI encoding
LogEscapeFill TLogEscape stack buffer with the (hexadecimal) chars of the input binary
LogEscapeFullReturns a text buffer with the (hexadecimal) chars of the input binary
LogEscapeFullReturns a text buffer with the (hexadecimal) chars of the input binary
MultiPartFormDataAddFieldEncode a field in a multipart array
MultiPartFormDataAddFileEncode a file in a multipart array
MultiPartFormDataDecodeDecode multipart/form-data POST request content into memory
MultiPartFormDataEncodeEncode multipart fields and files
MultiPartFormDataNewBoundUsed e.g. by MultiPartFormDataEncode and THttpMultiPartStream.Add
PluralWrite count number and append 's' (if needed) to form a plural English noun
RawBlobToBlobCreates a TEXT-encoded version of blob data from a memory data
RawBlobToBlobCreates a TEXT-encoded version of blob data from a RawBlob
RawByteStringArrayConcatFast concatenation of several AnsiStrings
RawByteStringToBytesCreates a TBytes from a RawByteString memory buffer
ResourceSynLZToRawByteStringCreates a RawByteString memory buffer from an SynLZ-compressed embedded resource
ResourceToRawByteStringCreates a RawByteString memory buffer from an embedded resource
SameFileContentCompare two files by content, reading them by blocks
ToVarInt32Convert an integer into a 32-bit variable-length integer buffer
ToVarInt64Convert a Int64 into a 64-bit variable-length integer buffer
ToVarStringConvert a RawUtf8 into an UTF-8 encoded variable-length buffer
ToVarUInt32Convert a cardinal into a 32-bit variable-length integer buffer
ToVarUInt32LengthReturn the number of bytes necessary to store a 32-bit variable-length integer
ToVarUInt32LengthWithDataReturn the number of bytes necessary to store some data with a its 32-bit variable-length integer length
ToVarUInt64Convert a UInt64 into a 64-bit variable-length integer buffer
UrlAppendAppend some encoded Name,Value pairs parameters to an existing URI
UrlDecodeDecode a UrlEncode() URI encoded parameter into its original value
UrlDecodeDecode a UrlEncode() URI encoded parameter into its original value
UrlDecodeCardinalDecode a specified parameter compatible with URI encoding into its original cardinal numerical value
UrlDecodeDoubleDecode a specified parameter compatible with URI encoding into its original floating-point value
UrlDecodeExtendedDecode a specified parameter compatible with URI encoding into its original floating-point value
UrlDecodeInt64Decode a specified parameter compatible with URI encoding into its original Int64 numerical value
UrlDecodeIntegerDecode a specified parameter compatible with URI encoding into its original integer numerical value
UrlDecodeNameDecode a UrlEncodeName() URI encoded network name into its original value
UrlDecodeNameDecode a UrlEncodeName() URI encoded network name into its original value
UrlDecodeNeedParametersReturns TRUE if all supplied parameters do exist in the URI encoded text
UrlDecodeNextNameDecode a URI-encoded Name from an input buffer
UrlDecodeNextNameValueDecode the next Name=Value&.... pair from input URI
UrlDecodeNextValueDecode a URI-encoded Value from an input buffer
UrlDecodeValueDecode a specified parameter compatible with URI encoding into its original textual value
UrlDecodeVarDecode a UrlEncode/UrlEncodeName() URI encoded string into its original value
UrlEncodeEncode a string as URI parameter encoding, i.e. ' ' as '+'
UrlEncodeEncode supplied parameters to be compatible with URI encoding
UrlEncodeEncode supplied parameters to be compatible with URI encoding
UrlEncodeEncode a string as URI parameter encoding, i.e. ' ' as '+'
UrlEncodeAppend a string as URI parameter encoding, i.e. ' ' as '+'
UrlEncodeAppend a string as URI parameter encoding, i.e. ' ' as '+'
UrlEncodeFullEncode a full URI with prefix and parameters
UrlEncodeNameEncode a string as URI network name encoding, i.e. ' ' as %20
UrlEncodeNameEncode a string as URI network name encoding, i.e. ' ' as %20
UrlEncodeNameAppend a string as URI network name encoding, i.e. ' ' as %20
UrlEncodeNameAppend a string as URI network name encoding, i.e. ' ' as %20

1.6.1. AddHtmlEscapeMarkdown

procedure AddHtmlEscapeMarkdown(W: TTextWriter; P: PUtf8Char; esc: TTextWriterHtmlEscape = [heEmojiToUtf8]);

Convert minimal Markdown text into proper HTML
- see https://enterprise.github.com/downloads/en/markdown-cheatsheet.pdf
- convert all #13#10 into <p>...</p>, *..* into <em>..</em>, **..** into <strong>..</strong>, `...` into <code>...</code>, backslash espaces \\ \* \_ and so on, [title](http://...) and detect plain http:// as <a href=...>
- create unordered lists from trailing * + - chars, blockquotes from trailing > char, and code line from 4 initial spaces
- as with default Markdown, won't escape HTML special chars (i.e. you can write plain HTML in the supplied text) unless esc is set otherwise
- only inline-style links and images are supported yet (not reference-style); tables aren't supported either


1.6.2. AddHtmlEscapeWiki

procedure AddHtmlEscapeWiki(W: TTextWriter; P: PUtf8Char; esc: TTextWriterHtmlEscape = [heHtmlEscape, heEmojiToUtf8]);

Convert some wiki-like text into proper HTML
- convert all #13#10 into <p>...</p>, *..* into <em>..</em>, +..+ into <strong>..</strong>, `..` into <code>..</code>, and http://... as <a href=http://...>
- escape any HTML special chars, and Emoji tags as specified with esc


1.6.3. Append999ToBuffer

function Append999ToBuffer(Buffer: PUtf8Char; Value: PtrUInt): PUtf8Char;

Fast add text conversion of 0-999 integer value into a given buffer
- warning: it won't check that Value is in 0-999 range
- up to 4 bytes may be written to the buffer (including #0 terminator)


1.6.4. AppendBuffersToRawUtf8

procedure AppendBuffersToRawUtf8(var Text: RawUtf8; const Buffers: array of PUtf8Char);

Fast add some characters to a RawUtf8 string
- faster than Text := Text+RawUtf8(Buffers[0])+RawUtf8(Buffers[0])+...
- will handle up to 64 Buffers[] - raise an ESynException on too many Buffers


1.6.5. AppendBufferToBuffer

function AppendBufferToBuffer(Buffer: PUtf8Char; Text: pointer; Len: PtrInt): PUtf8Char;

Fast add some characters from ane buffer into another buffer
- warning: the Buffer should contain enough space to store the Text, otherwise you may encounter buffer overflows and random memory errors


1.6.6. AppendBufferToRawByteString

procedure AppendBufferToRawByteString(var Content: RawByteString; const Buffer; BufferLen: PtrInt); overload;

Just redirect to mormot.core.text Append(...) overloads


1.6.7. AppendCharOnceToRawUtf8

procedure AppendCharOnceToRawUtf8(var Text: RawUtf8; Ch: AnsiChar);

Fast add one character to a RawUtf8 string, if not already present
- avoid a temporary memory allocation of a string, so faster alternative to

 if (Text<>'') and (Text[length(Text)]<>Ch) then Text := Text + ch;

1.6.8. AppendRawUtf8ToBuffer

function AppendRawUtf8ToBuffer(Buffer: PUtf8Char; const Text: RawUtf8): PUtf8Char;

Fast add some characters from a RawUtf8 string into a given buffer
- warning: the Buffer should contain enough space to store the Text, otherwise you may encounter buffer overflows and random memory errors


1.6.9. AppendUInt32ToBuffer

function AppendUInt32ToBuffer(Buffer: PUtf8Char; Value: PtrUInt): PUtf8Char;

Fast add text conversion of a 32-bit signed integer value into a given buffer
- warning: the Buffer should contain enough space to store the text, otherwise you may encounter buffer overflows and random memory errors


1.6.10. AreUrlValid

function AreUrlValid(const Url: array of RawUtf8): boolean;

Checks if the supplied UTF-8 text values don't need URI encoding
- returns TRUE if all its chars of all strings are non-void plain ASCII-7 RFC compatible identifiers (0..9a..zA..Z-_.~)


1.6.11. AsciiToBaudot

function AsciiToBaudot(P: PAnsiChar; len: PtrInt): RawByteString; overload;

Convert some ASCII-7 text into binary, using Emile Baudot code
- as used in telegraphs, covering #10 #13 #32 a-z 0-9 - ' , ! : ( + ) $ ? @ . / ; charset, following a custom static-huffman-like encoding with 5-bit masks
- any upper case char will be converted into lowercase during encoding
- other characters (e.g. UTF-8 accents, or controls chars) will be ignored
- resulting binary will consume 5 (or 10) bits per character
- reverse of the BaudotToAscii() function
- the "baud" symbol rate measurement comes from Emile's name ;)


1.6.12. AsciiToBaudot

function AsciiToBaudot(const Text: RawUtf8): RawByteString; overload;

Convert some ASCII-7 text into binary, using Emile Baudot code
- as used in telegraphs, covering #10 #13 #32 a-z 0-9 - ' , ! : ( + ) $ ? @ . / ; charset, following a custom static-huffman-like encoding with 5-bit masks
- any upper case char will be converted into lowercase during encoding
- other characters (e.g. UTF-8 accents, or controls chars) will be ignored
- resulting binary will consume 5 (or 10) bits per character
- reverse of the BaudotToAscii() function
- the "baud" symbol rate measurement comes from Emile's name ;)


1.6.13. Base32ToBin

function Base32ToBin(B32: PAnsiChar; B32Len: integer): RawByteString; overload;

Conversion from Base32 encoded text into a binary string
- RFC4648 Base32 is defined as upper alphanumeric without misleading 0O 1I 8B
- returns '' if input was not valid Base32 encoded


1.6.14. Base32ToBin

function Base32ToBin(const base32: RawUtf8): RawByteString; overload;

Conversion from Base32 encoded text into a binary string
- RFC4648 Base32 is defined as upper alphanumeric without misleading 0O 1I 8B
- returns '' if input was not valid Base32 encoded


1.6.15. Base58ToBin

function Base58ToBin(B58: PAnsiChar; B58Len: integer; var Dest: TSynTempBuffer): integer; overload;

Conversion from Base58 encoded text into a binary buffer
- Bitcoin' Base58 was defined as alphanumeric chars without misleading 0O I1
- Base58 is much slower than Base64, performing in O(n^2) instead of O(n), and should not be used on big buffers
- returns the number of decoded chars encoded into Dest.buf
- caller should call Dest.Done once it is finished with the output binary


1.6.16. Base58ToBin

function Base58ToBin(B58: PAnsiChar; B58Len: integer): RawByteString; overload;

Conversion from Base58 encoded text into a binary string
- Bitcoin' Base58 was defined as alphanumeric chars without misleading 0O I1
- Base58 is much slower than Base64, and should not be used on big buffers
- returns '' if input was not valid Base58 encoded


1.6.17. Base58ToBin

function Base58ToBin(const base58: RawUtf8): RawByteString; overload;

Conversion from Base58 encoded text into a binary string
- Bitcoin' Base58 was defined as alphanumeric chars without misleading 0O I1
- Base58 is much slower than Base64, and should not be used on big buffers
- returns '' if input was not valid Base58 encoded


1.6.18. Base64Decode

function Base64Decode(sp, rp: PAnsiChar; len: PtrInt): boolean;

Direct low-level decoding of a Base64 encoded buffer
- here len is the number of 4 chars chunks in sp input
- deprecated low-level function: use Base64ToBin/Base64ToBinSafe instead


1.6.19. Base64Encode

procedure Base64Encode(rp, sp: PAnsiChar; len: cardinal);

Raw function for efficient binary to Base64 encoding
- just a wrapper around Base64EncodeMain() + Base64EncodeTrailing()


1.6.20. Base64EncodeTrailing

procedure Base64EncodeTrailing(rp, sp: PAnsiChar; len: cardinal);

Raw function for efficient binary to Base64 encoding of the last bytes
- don't use this function, but rather the BinToBase64() overloaded functions


1.6.21. Base64MagicCheckAndDecode

function Base64MagicCheckAndDecode(Value: PUtf8Char; ValueLen: integer; var Blob: RawByteString): boolean; overload;

Check and decode '\uFFF0base64encodedbinary' content into binary
- this method will check the supplied value to match the expected JSON_BASE64_MAGIC_C pattern, decode and set Blob and return TRUE


1.6.22. Base64MagicCheckAndDecode

function Base64MagicCheckAndDecode(Value: PUtf8Char; var Blob: TSynTempBuffer; ValueLen: integer = 0): boolean; overload;

Check and decode '\uFFF0base64encodedbinary' content into binary
- this method will check the supplied value to match the expected JSON_BASE64_MAGIC_C pattern, decode and set Blob and return TRUE


1.6.23. Base64MagicCheckAndDecode

function Base64MagicCheckAndDecode(Value: PUtf8Char; var Blob: RawByteString): boolean; overload;

Check and decode '\uFFF0base64encodedbinary' content into binary
- this method will check the supplied value to match the expected JSON_BASE64_MAGIC_C pattern, decode and set Blob and return TRUE


1.6.24. Base64MagicDecode

procedure Base64MagicDecode(var ParamValue: RawUtf8);

Just a wrapper around Base64ToBin() for in-place decode of JSON_BASE64_MAGIC_C '\uFFF0base64encodedbinary' content into binary
- input ParamValue shall have been checked to match the expected pattern


1.6.25. Base64MagicToBlob

procedure Base64MagicToBlob(Base64: PUtf8Char; var result: RawUtf8);

Convert a Base64-encoded content into binary hexadecimal ready for SQL
- returns e.g. X'53514C697465'


1.6.26. Base64MagicTryAndDecode

function Base64MagicTryAndDecode(Value: PUtf8Char; ValueLen: integer; var Blob: RawByteString): boolean;

Decode '\uFFF0base64encodedbinary' or 'base64encodedbinary' into binary
- same as Base64MagicCheckAndDecode(), but will detect and ignore the magic and not require it


1.6.27. Base64ToBin

function Base64ToBin(base64, bin: PAnsiChar; base64len, binlen: PtrInt ; nofullcheck: boolean = true ): boolean; overload;

Fast conversion from Base64 encoded text into binary data
- returns TRUE on success, FALSE if base64 does not match binlen
- nofullcheck is deprecated and not used any more, since nofullcheck=false is now processed with no performance cost


1.6.28. Base64ToBin

function Base64ToBin(const base64: RawByteString; bin: PAnsiChar; binlen: PtrInt ; nofullcheck: boolean = true ): boolean; overload;

Fast conversion from Base64 encoded text into binary data
- returns TRUE on success, FALSE if base64 does not match binlen
- nofullcheck is deprecated and not used any more, since nofullcheck=false is now processed with no performance cost


1.6.29. Base64ToBin

function Base64ToBin(sp: PAnsiChar; len: PtrInt; var Blob: TSynTempBuffer): boolean; overload;

Fast conversion from Base64 encoded text into binary data
- returns TRUE on success, FALSE if sp/len buffer was invvalid


1.6.30. Base64ToBin

function Base64ToBin(const s: RawByteString): RawByteString; overload;

Fast conversion from Base64 encoded text into binary data
- is now just an alias to Base64ToBinSafe() overloaded function
- returns '' if s was not a valid Base64-encoded input


1.6.31. Base64ToBin

function Base64ToBin(sp: PAnsiChar; len: PtrInt): RawByteString; overload;

Fast conversion from Base64 encoded text into binary data
- is now just an alias to Base64ToBinSafe() overloaded function
- returns '' if sp/len buffer was not a valid Base64-encoded input


1.6.32. Base64ToBin

function Base64ToBin(sp: PAnsiChar; len: PtrInt; var data: RawByteString): boolean; overload;

Fast conversion from Base64 encoded text into binary data
- is now just an alias to Base64ToBinSafe() overloaded function
- returns false and data='' if sp/len buffer was invalid


1.6.33. Base64ToBinLength

function Base64ToBinLength(sp: PAnsiChar; len: PtrInt): PtrInt;

Retrieve the expected undecoded length of a Base64 encoded buffer
- here len is the number of bytes in sp


1.6.34. Base64ToBinLengthSafe

function Base64ToBinLengthSafe(sp: PAnsiChar; len: PtrInt): PtrInt;

Retrieve the expected undecoded length of a Base64 encoded buffer
- here len is the number of bytes in sp
- will check supplied text is a valid Base64 encoded stream


1.6.35. Base64ToBinSafe

function Base64ToBinSafe(const s: RawByteString): RawByteString; overload;

Fast conversion from Base64 encoded text into binary data
- will check supplied text is a valid Base64 encoded stream


1.6.36. Base64ToBinSafe

function Base64ToBinSafe(sp: PAnsiChar; len: PtrInt): RawByteString; overload;

Fast conversion from Base64 encoded text into binary data
- will check supplied text is a valid Base64 encoded stream


1.6.37. Base64ToBinSafe

function Base64ToBinSafe(sp: PAnsiChar; len: PtrInt; var data: RawByteString): boolean; overload;

Fast conversion from Base64 encoded text into binary data
- will check supplied text is a valid Base64 encoded stream


1.6.38. Base64ToBinSafe

function Base64ToBinSafe(sp: PAnsiChar; len: PtrInt; out data: TBytes): boolean; overload;

Fast conversion from Base64 encoded text into binary data
- will check supplied text is a valid Base64 encoded stream


1.6.39. Base64ToBinTrim

function Base64ToBinTrim(const s: RawByteString): RawByteString;

Conversion from Base64 encoded text into binary data, ignoring spaces
- returns '' if s was not a valid Base64-encoded input once spaces are trimmed
- consider PemToDer() from mormot.crypt.secure if you need to read PEM content


1.6.40. Base64ToUri

procedure Base64ToUri(var base64: RawUtf8);

Conversion from any Base64 encoded value into URI-compatible encoded text
- warning: will modify the supplied base64 string in-place
- in comparison to Base64 standard encoding, will trim any right-sided '=' unsignificant characters, and replace '+' or '/' by '_' or '-'


1.6.41. Base64uriDecode

function Base64uriDecode(sp, rp: PAnsiChar; len: PtrInt): boolean;

Direct low-level decoding of a Base64-URI encoded buffer
- the buffer is expected to be at least Base64uriToBinLength() bytes long
- returns true if the supplied sp[] buffer has been successfully decoded into rp[] - will break at any invalid character, so is always safe to use
- in comparison to Base64 standard encoding, will trim any right-sided '=' unsignificant characters, and replace '+' or '/' by '_' or '-'
- you should better not use this, but Base64uriToBin() overloaded functions


1.6.42. Base64uriEncode

procedure Base64uriEncode(rp, sp: PAnsiChar; len: cardinal);

Low-level conversion from a binary buffer into Base64-like URI-compatible encoded text
- you should rather use the overloaded BinToBase64uri() functions


1.6.43. Base64uriToBin

function Base64uriToBin(sp: PAnsiChar; len: PtrInt; var temp: TSynTempBuffer): boolean; overload;

Fast conversion from Base64-URI encoded text into binary data
- caller should always execute temp.Done when finished with the data
- in comparison to Base64 standard encoding, will trim any right-sided '=' unsignificant characters, and replace '+' or '/' by '_' or '-'


1.6.44. Base64uriToBin

function Base64uriToBin(sp: PAnsiChar; len: PtrInt; var bin: RawByteString): boolean; overload;

Fast conversion from Base64-URI encoded text into binary data
- in comparison to Base64 standard encoding, will trim any right-sided '=' unsignificant characters, and replace '+' or '/' by '_' or '-'


1.6.45. Base64uriToBin

function Base64uriToBin(const s: RawByteString): RawByteString; overload;

Fast conversion from Base64-URI encoded text into binary data
- in comparison to Base64 standard encoding, will trim any right-sided '=' unsignificant characters, and replace '+' or '/' by '_' or '-'


1.6.46. Base64uriToBin

function Base64uriToBin(const base64: RawByteString; bin: PAnsiChar; binlen: PtrInt): boolean; overload;

Fast conversion from Base64-URI encoded text into binary data
- in comparison to Base64 standard encoding, will trim any right-sided '=' unsignificant characters, and replace '+' or '/' by '_' or '-'
- will check supplied text is a valid Base64-URI encoded stream


1.6.47. Base64uriToBin

function Base64uriToBin(base64, bin: PAnsiChar; base64len, binlen: PtrInt): boolean; overload;

Fast conversion from Base64-URI encoded text into binary data
- in comparison to Base64 standard encoding, will trim any right-sided '=' unsignificant characters, and replace '+' or '/' by '_' or '-'
- will check supplied text is a valid Base64-URI encoded stream


1.6.48. Base64uriToBin

function Base64uriToBin(sp: PAnsiChar; len: PtrInt): RawByteString; overload;

Fast conversion from Base64-URI encoded text into binary data
- in comparison to Base64 standard encoding, will trim any right-sided '=' unsignificant characters, and replace '+' or '/' by '_' or '-'


1.6.49. Base64uriToBinLength

function Base64uriToBinLength(len: PtrInt): PtrInt;

Retrieve the expected undecoded length of a Base64-URI encoded buffer
- here len is the number of bytes in sp
- in comparison to Base64 standard encoding, will trim any right-sided '=' unsignificant characters, and replace '+' or '/' by '_' or '-'


1.6.50. BaudotToAscii

function BaudotToAscii(const Baudot: RawByteString): RawUtf8; overload;

Convert some Baudot code binary, into ASCII-7 text
- reverse of the AsciiToBaudot() function
- any uppercase character would be decoded as lowercase - and some characters may have disapeared outside of a-z 0-9 - ' , ! : ( + ) $ ? @ . / ; range
- the "baud" symbol rate measurement comes from Emile's name ;)


1.6.51. BaudotToAscii

function BaudotToAscii(Baudot: PByteArray; len: PtrInt): RawUtf8; overload;

Convert some Baudot code binary, into ASCII-7 text
- reverse of the AsciiToBaudot() function
- any uppercase character would be decoded as lowercase - and some characters may have disapeared outside of a-z 0-9 - ' , ! : ( + ) $ ? @ . / ; range
- the "baud" symbol rate measurement comes from Emile's name ;)


1.6.52. BinToBase32

procedure BinToBase32(Bin: PByteArray; Dest: PAnsiChar; BinLen: PtrInt; b32enc: PAnsiChar); overload;

Conversion from a binary buffer into Base32 encoded text buffer
- default b32enc is RFC4648 upper alphanumeric without misleading 0O 1I 8B


1.6.53. BinToBase32

function BinToBase32(Bin: PAnsiChar; BinLen: PtrInt): RawUtf8; overload;

Conversion from a binary buffer into Base32 encoded text as RawUtf8
- RFC4648 Base32 is defined as upper alphanumeric without misleading 0O 1I 8B


1.6.54. BinToBase32

function BinToBase32(const Bin: RawByteString): RawUtf8; overload;

Conversion from a binary buffer into Base32 encoded text as RawUtf8
- RFC4648 Base32 is defined as upper alphanumeric without misleading 0O 1I 8B


1.6.55. BinToBase32Length

function BinToBase32Length(BinLen: cardinal): cardinal;

Compute the length resulting of Base32 encoding of a binary buffer
- RFC4648 Base32 is defined as upper alphanumeric without misleading 0O 1I 8B


1.6.56. BinToBase58

function BinToBase58(const Bin: RawByteString): RawUtf8; overload;

Conversion from a binary buffer into Base58 encoded text as RawUtf8
- Bitcoin' Base58 was defined as alphanumeric chars without misleading 0O I1
- Base58 is much slower than Base64, performing in O(n^2) instead of O(n), and should not be used on big buffers


1.6.57. BinToBase58

function BinToBase58(Bin: PAnsiChar; BinLen: integer): RawUtf8; overload;

Conversion from a binary buffer into Base58 encoded text as RawUtf8
- Bitcoin' Base58 was defined as alphanumeric chars without misleading 0O I1
- Base58 is much slower than Base64, performing in O(n^2) instead of O(n), and should not be used on big buffers


1.6.58. BinToBase58

function BinToBase58(Bin: PAnsiChar; BinLen: integer; var Dest: TSynTempBuffer): integer; overload;

Conversion from a binary buffer into Base58 encoded text as TSynTempBuffer
- Bitcoin' Base58 was defined as alphanumeric chars without misleading 0O I1
- Base58 is much slower than Base64, performing in O(n^2) instead of O(n), and should not be used on big buffers
- returns the number of encoded chars encoded into Dest.buf
- caller should call Dest.Done once it is finished with the output text


1.6.59. BinToBase64

function BinToBase64(Bin: PAnsiChar; BinBytes: integer): RawUtf8; overload;

Fast conversion from binary data into Base64 encoded UTF-8 text


1.6.60. BinToBase64

function BinToBase64(const s: RawByteString): RawUtf8; overload;

Fast conversion from binary data into Base64 encoded UTF-8 text


1.6.61. BinToBase64

function BinToBase64(const data, Prefix, Suffix: RawByteString; WithMagic: boolean): RawUtf8; overload;

Fast conversion from binary data into prefixed/suffixed Base64 encoded UTF-8 text
- with optional JSON_BASE64_MAGIC_C prefix (UTF-8 encoded \uFFF0 special code)
- may use AVX2 optimized asm (10GB/s) on FPC x86_64


1.6.62. BinToBase64Length

function BinToBase64Length(len: PtrUInt): PtrUInt;

Retrieve the expected encoded length after Base64 process


1.6.63. BinToBase64Line

function BinToBase64Line(sp: PAnsiChar; len: PtrUInt; const Prefix: RawUtf8 = ''; const Suffix: RawUtf8 = ''): RawUtf8;

Fast conversion from binary into prefixed/suffixed Base64 with 64 chars per line


1.6.64. BinToBase64Short

function BinToBase64Short(Bin: PAnsiChar; BinBytes: integer): ShortString; overload;

Fast conversion from a small binary data into Base64 encoded UTF-8 text


1.6.65. BinToBase64Short

function BinToBase64Short(const s: RawByteString): ShortString; overload;

Fast conversion from a small binary data into Base64 encoded UTF-8 text


1.6.66. BinToBase64uri

function BinToBase64uri(Bin: PAnsiChar; BinBytes: integer): RawUtf8; overload;

Fast conversion from a binary buffer into Base64-like URI-compatible encoded text
- in comparison to Base64 standard encoding, will trim any right-sided '=' unsignificant characters, and replace '+' or '/' by '_' or '-'


1.6.67. BinToBase64uri

function BinToBase64uri(const s: RawByteString): RawUtf8; overload;

Fast conversion from binary data into Base64-like URI-compatible encoded text
- in comparison to Base64 standard encoding, will trim any right-sided '=' unsignificant characters, and replace '+' or '/' by '_' or '-'


1.6.68. BinToBase64uriLength

function BinToBase64uriLength(len: PtrUInt): PtrUInt;

Retrieve the expected encoded length after Base64-URI process
- in comparison to Base64 standard encoding, will trim any right-sided '=' unsignificant characters, and replace '+' or '/' by '_' or '-'


1.6.69. BinToBase64uriShort

function BinToBase64uriShort(Bin: PAnsiChar; BinBytes: integer): ShortString;

Fast conversion from a binary buffer into Base64-like URI-compatible encoded ShortString
- in comparison to Base64 standard encoding, will trim any right-sided '=' unsignificant characters, and replace '+' or '/' by '_' or '-'
- returns '' if BinBytes void or too big for the resulting ShortString


1.6.70. BinToBase64WithMagic

function BinToBase64WithMagic(const data: RawByteString): RawUtf8; overload;

Fast conversion from binary data into Base64 encoded UTF-8 text with JSON_BASE64_MAGIC_C prefix (UTF-8 encoded \uFFF0 special code)


1.6.71. BinToBase64WithMagic

procedure BinToBase64WithMagic(Data: pointer; DataLen: integer; var Result: RawUtf8); overload;

Fast conversion from binary data into Base64 encoded UTF-8 text with JSON_BASE64_MAGIC_C prefix (UTF-8 encoded \uFFF0 special code)


1.6.72. BinToBase64WithMagic

function BinToBase64WithMagic(Data: pointer; DataLen: integer): RawUtf8; overload;

Fast conversion from binary data into Base64 encoded UTF-8 text with JSON_BASE64_MAGIC_C prefix (UTF-8 encoded \uFFF0 special code)


1.6.73. BinToHumanHex

function BinToHumanHex(Data: PByte; Len: integer; PerLine: integer = 16; LeftTab: integer = 0; SepChar: AnsiChar = ':'): RawUtf8; overload;

Generate some 'xx:xx:xx:xx' output buffer with left and right margins
- used e.g. by ParsedToText() to output X509 public key content in PeerInfo


1.6.74. BinToHumanHex

procedure BinToHumanHex(W: TTextWriter; Data: PByte; Len: integer; PerLine: integer = 16; LeftTab: integer = 0; SepChar: AnsiChar = ':'); overload;

Generate some 'xx:xx:xx:xx' output buffer with left and right margins


1.6.75. BinToSource

function BinToSource(const ConstName, Comment: RawUtf8; const Data: RawByteString; PerLine: integer = 16; const Suffix: RawUtf8 = ''): RawUtf8; overload;

Generate some pascal source code holding some data binary as constant


1.6.76. BinToSource

function BinToSource(const ConstName, Comment: RawUtf8; Data: pointer; Len: integer; PerLine: integer = 16; const Suffix: RawUtf8 = ''): RawUtf8; overload;

Generate some pascal source code holding some data binary as constant
- can store sensitive information (e.g. certificates) within the executable
- generates a source code snippet of the following format:

 const
   // Comment
   ConstName: array[0..2] of byte = (
     $01, $02, $03);

1.6.77. BinToSource

procedure BinToSource(Dest: TTextWriter; const ConstName, Comment: RawUtf8; Data: pointer; Len: integer; PerLine: integer = 16); overload;

Generate some pascal source code holding some data binary as constant
- can store sensitive information (e.g. certificates) within the executable
- generates a source code snippet of the following format:

 const
   // Comment
   ConstName: array[0..2] of byte = (
     $01, $02, $03);

1.6.78. BlobToBytes

function BlobToBytes(P: PUtf8Char): TBytes;

Create a TBytes from TEXT-encoded blob data
- blob data can be encoded as SQLite3 BLOB literals (X'53514C697465' e.g.) or or Base64 encoded content ('\uFFF0base64encodedbinary') or plain TEXT


1.6.79. BlobToRawBlob

procedure BlobToRawBlob(P: PUtf8Char; var result: RawBlob; Len: integer = 0); overload;

Fill a RawBlob from TEXT-encoded blob data
- blob data can be encoded as SQLite3 BLOB literals (X'53514C697465' e.g.) or or Base64 encoded content ('\uFFF0base64encodedbinary') or plain TEXT


1.6.80. BlobToRawBlob

function BlobToRawBlob(const Blob: RawByteString): RawBlob; overload;

Fill a RawBlob from TEXT-encoded blob data
- blob data can be encoded as SQLite3 BLOB literals (X'53514C697465' e.g.) or or Base64 encoded content ('\uFFF0base64encodedbinary') or plain TEXT


1.6.81. BlobToRawBlob

function BlobToRawBlob(P: PUtf8Char; Len: integer = 0): RawBlob; overload;

Fill a RawBlob from TEXT-encoded blob data
- blob data can be encoded as SQLite3 BLOB literals (X'53514C697465' e.g.) or or Base64 encoded content ('\uFFF0base64encodedbinary') or plain TEXT


1.6.82. BlobToStream

function BlobToStream(P: PUtf8Char): TStream;

Create a memory stream from TEXT-encoded blob data
- blob data can be encoded as SQLite3 BLOB literals (X'53514C697465' e.g.) or or Base64 encoded content ('\uFFF0base64encodedbinary') or plain TEXT
- the caller must free the stream instance after use


1.6.83. BytesToRawByteString

procedure BytesToRawByteString(const bytes: TBytes; out buf: RawByteString);

Creates a RawByteString memory buffer from a TBytes content


1.6.84. ContentToShort

function ContentToShort(const source: RawByteString): ShortString;

If source is not UTF-8 calls EscapeToShort, otherwise return it directly


1.6.85. EmojiFromDots

function EmojiFromDots(const text: RawUtf8): RawUtf8; overload;

Conversion of github/Markdown :identifiers: into UTF-8 Emoji sequences


1.6.86. EmojiFromDots

procedure EmojiFromDots(P: PUtf8Char; W: TTextWriter); overload;

Low-level conversion of github/Markdown :identifiers: into UTF-8 Emoji sequences


1.6.87. EmojiFromText

function EmojiFromText(P: PUtf8Char; len: PtrInt): TEmoji;

Recognize github/Markdown compatible text of Emojis
- for instance 'sunglasses' text buffer will return eSunglasses
- returns eNone if no case-insensitive match was found


1.6.88. EmojiParseDots

function EmojiParseDots(var P: PUtf8Char; W: TTextWriter = nil): TEmoji;

Low-level parser of github/Markdown compatible text of Emojis
- supplied P^ should point to ':'
- will append the recognized UTF-8 Emoji if P contains e.g. :joy: or :)
- will append ':' if no Emoji text is recognized, and return eNone
- will try both EMOJI_AFTERDOTS[] and EMOJI_RTTI[] reference set
- if W is nil, won't append anything, but just return the recognized TEmoji


1.6.89. EmojiToDots

function EmojiToDots(const text: RawUtf8): RawUtf8; overload;

Conversion of UTF-8 Emoji sequences into github/Markdown :identifiers:


1.6.90. EmojiToDots

procedure EmojiToDots(P: PUtf8Char; W: TTextWriter); overload;

Low-level conversion of UTF-8 Emoji sequences into github/Markdown :identifiers:


1.6.91. EscapeBuffer

function EscapeBuffer(s: PAnsiChar; slen: integer; d: PAnsiChar; dmax: integer): PAnsiChar;

Low-level fast conversion from binary data to escaped text
- non printable characters will be written as $xx hexadecimal codes
- will be #0 terminated, with '...' characters trailing on dmax overflow
- ensure the destination buffer contains at least dmax bytes, which is always the case when using LogEscape() and its local TLogEscape variable


1.6.92. EscapeToShort

function EscapeToShort(source: PAnsiChar; sourcelen: integer): ShortString; overload;

Fill a ShortString with the (hexadecimal) chars of the input text/binary


1.6.93. EscapeToShort

function EscapeToShort(const source: RawByteString): ShortString; overload;

Fill a ShortString with the (hexadecimal) chars of the input text/binary


1.6.94. FromVarBlob

function FromVarBlob(Data: PByte): TValueResult;

Retrieve pointer and length to a variable-length text/blob buffer


1.6.95. FromVarInt32

function FromVarInt32(var Source: PByte): integer;

Convert a 32-bit variable-length integer buffer into an integer
- decode negative values from cardinal two-complement, i.e. 0=0,1=1,2=-1,3=2,4=-2...


1.6.96. FromVarInt64

function FromVarInt64(var Source: PByte): Int64;

Convert a 64-bit variable-length integer buffer into a Int64


1.6.97. FromVarInt64Value

function FromVarInt64Value(Source: PByte): Int64;

Convert a 64-bit variable-length integer buffer into a Int64
- this version won't update the Source pointer


1.6.98. FromVarString

procedure FromVarString(var Source: PByte; var Value: RawByteString; CodePage: integer); overload;

Retrieve a variable-length text buffer
- this overloaded function will set the supplied code page to the AnsiString


1.6.99. FromVarString

function FromVarString(var Source: PByte; SourceMax: PByte; var Value: RawByteString; CodePage: integer): boolean; overload;

Retrieve a variable-length text buffer
- this overloaded function will set the supplied code page to the AnsiString and will also check for the SourceMax end of buffer
- returns TRUE on success, or FALSE on any buffer overload detection


1.6.100. FromVarString

procedure FromVarString(var Source: PByte; var Value: RawUtf8); overload;

Retrieve a variable-length UTF-8 encoded text buffer in a newly allocation RawUtf8


1.6.101. FromVarString

function FromVarString(var Source: PByte; SourceMax: PByte): RawUtf8; overload;

Safe retrieve a variable-length UTF-8 encoded text buffer in a newly allocation RawUtf8
- supplied SourceMax value will avoid any potential buffer overflow


1.6.102. FromVarString

function FromVarString(var Source: PByte): RawUtf8; overload;

Retrieve a variable-length UTF-8 encoded text buffer in a newly allocation RawUtf8


1.6.103. FromVarString

function FromVarString(var Source: PByte; SourceMax: PByte; var Value: TSynTempBuffer): boolean; overload;

Retrieve a variable-length UTF-8 encoded text buffer in a temporary buffer
- caller should call Value.Done after use of the Value.buf memory
- this overloaded function will also check for the SourceMax end of buffer, returning TRUE on success, or FALSE on any buffer overload detection


1.6.104. FromVarString

procedure FromVarString(var Source: PByte; var Value: TSynTempBuffer); overload;

Retrieve a variable-length UTF-8 encoded text buffer in a temporary buffer
- caller should call Value.Done after use of the Value.buf memory
- this overloaded function would include a #0 terminator, so Value.buf could be parsed as a valid PUtf8Char buffer (e.g. containing JSON)


1.6.105. FromVarUInt32

function FromVarUInt32(var Source: PByte; SourceMax: PByte; out Value: cardinal): boolean; overload;

Convert a 32-bit variable-length integer buffer into a cardinal
- will call FromVarUInt32() if SourceMax=nil, or FromVarUInt32Safe() if set
- returns false on error, true if Value has been set properly


1.6.106. FromVarUInt32

function FromVarUInt32(var Source: PByte): cardinal; overload;

Convert a 32-bit variable-length integer buffer into a cardinal
- fast inlined process for any number < 128
- use overloaded FromVarUInt32() or FromVarUInt32Safe() with a SourceMax pointer to avoid any potential buffer overflow
- use FromVarUInt32Big() is the content is likely to be >= 128


1.6.107. FromVarUInt32Big

function FromVarUInt32Big(var Source: PByte): cardinal;

Convert a 32-bit variable-length integer buffer into a cardinal
- this version could be called if number is likely to be > $7f, so if inlining the first byte won't make any benefit


1.6.108. FromVarUInt32High

function FromVarUInt32High(var Source: PByte): cardinal;

Convert a 32-bit variable-length integer buffer into a cardinal
- this version must be called if Source^ has already been checked to be > $7f


1.6.109. FromVarUInt32Safe

function FromVarUInt32Safe(Source, SourceMax: PByte; out Value: cardinal): PByte;

Safely convert a 32-bit variable-length integer buffer into a cardinal
- slower but safer process checking out of boundaries memory access in Source
- SourceMax is expected to be not nil, and to point to the first byte just after the Source memory buffer
- returns nil on error, or point to next input data on successful decoding


1.6.110. FromVarUInt32Up128

function FromVarUInt32Up128(var Source: PByte): cardinal;

Convert a 32-bit variable-length integer buffer into a cardinal
- used e.g. when inlining FromVarUInt32()
- this version must be called if Source^ has already been checked to be > $7f

 result := Source^;
 inc(Source);
 if result>$7f then
   result := (result and $7F) or FromVarUInt32Up128(Source);

1.6.111. FromVarUInt64

function FromVarUInt64(var Source: PByte; SourceMax: PByte; out Value: Qword): boolean; overload;

Convert a 64-bit variable-length integer buffer into a UInt64
- will call FromVarUInt64() if SourceMax=nil, or FromVarUInt64Safe() if set
- returns false on error, true if Value has been set properly


1.6.112. FromVarUInt64

function FromVarUInt64(var Source: PByte): QWord; overload;

Convert a 64-bit variable-length integer buffer into a UInt64


1.6.113. FromVarUInt64Safe

function FromVarUInt64Safe(Source, SourceMax: PByte; out Value: QWord): PByte;

Safely convert a 64-bit variable-length integer buffer into a UInt64
- slower but safer process checking out of boundaries memory access in Source
- SourceMax is expected to be not nil, and to point to the first byte just after the Source memory buffer
- returns nil on error, or point to next input data on successful decoding


1.6.114. GetJpegSize

function GetJpegSize(jpeg: PAnsiChar; len: PtrInt; out Height, Width, Bits: integer): boolean; overload;

Fast guess of the size, in pixels, of a JPEG memory buffer
- will only scan for basic JPEG structure, up to the StartOfFrame (SOF) chunk
- returns TRUE if the buffer is likely to be a JPEG picture, and set the Height + Width + Bits variable with its dimensions - but there may be false positive recognition, and no waranty that the memory buffer is a valid JPEG
- returns FALSE if the buffer does not have any expected SOI/SOF markers


1.6.115. GetMimeContentType

function GetMimeContentType(Content: pointer; Len: PtrInt; const FileName: TFileName = ''; const DefaultContentType: RawUtf8 = BINARY_CONTENT_TYPE; Mime: PMimeType = nil): RawUtf8;

Retrieve the MIME content type from its file name or a supplied binary buffer
- will first check for known file extensions, then inspect the binary content
- return the MIME type, ready to be appended to a 'Content-Type: ' HTTP header
- default is DefaultContentType or 'application/octet-stream' (BINARY_CONTENT_TYPE) or 'application/fileextension' if FileName was specified
- see @http://en.wikipedia.org/wiki/Internet_media_type for most common values


1.6.116. GetMimeContentTypeFromBuffer

function GetMimeContentTypeFromBuffer(Content: pointer; Len: PtrInt; const DefaultContentType: RawUtf8; Mime: PMimeType = nil): RawUtf8;

Retrieve the MIME content type from a supplied binary buffer
- inspect the first bytes, to guess from standard known headers
- return the MIME type, ready to be appended to a 'Content-Type: ' HTTP header
- returns DefaultContentType if the binary buffer has an unknown layout


1.6.117. GetMimeContentTypeFromExt

function GetMimeContentTypeFromExt(const FileName: TFileName; FileExt: PRawUtf8 = nil): TMimeType;

MtUnknown mtPng mtGif mtTiff mtJpg mtBmp mtDoc mtPpt mtXls mtHtml mtCss mtJS RFC 9239 mtXIcon mtFont RFC 8081 mtText mtSvg mtXml mtWebp mtManifest mtJson mtOgg RFC 5334 mtMp4 RFC 4337 6381 mtMp2 mtMpeg RFC 3003 mtH264 RFC 6184 mtWma mtWmv mtAvi mtGzip mtWebm mtRar mt7z mtZip mtBz2 mtPdf mtSQlite3 mtXcomp mtDicom retrieve the MIME content type from its file name


1.6.118. GetMimeContentTypeFromMemory

function GetMimeContentTypeFromMemory(Content: pointer; Len: PtrInt): TMimeType;

Retrieve the MIME content type from a supplied binary buffer


1.6.119. GetMimeContentTypeHeader

function GetMimeContentTypeHeader(const Content: RawByteString; const FileName: TFileName = ''; Mime: PMimeType = nil): RawUtf8;

Retrieve the HTTP header for MIME content type from a supplied binary buffer
- just append HEADER_CONTENT_TYPE and GetMimeContentType() result
- can be used as such:

  Call.OutHead := GetMimeContentTypeHeader(Call.OutBody,aFileName);

1.6.120. GetMimeTypeFromExt

function GetMimeTypeFromExt(const Ext: RawUtf8): TMimeType;

Retrieve the MIME content type from its file extension text (without '.')


1.6.121. GetStreamBuffer

function GetStreamBuffer(S: TStream): pointer;

Retrieve the memory buffer of a TCustomMemoryStream/TRawByteStringStream
- returns nil if the instance is not of those classes


1.6.122. GotoNextVarInt

function GotoNextVarInt(Source: PByte): pointer;

Jump a value in the 32-bit or 64-bit variable-length integer buffer


1.6.123. GotoNextVarString

function GotoNextVarString(Source: PByte): pointer;

Jump a value in variable-length text buffer


1.6.124. HashFile

function HashFile(const FileName: TFileName; Hasher: THasher = nil): cardinal; overload;

Compute the 32-bit default hash of a file content
- you can specify your own hashing function if DefaultHasher is not what you expect


1.6.125. HashFileCrc32c

function HashFileCrc32c(const FileName: TFileName): RawUtf8;

Compute the crc32c checksum of a given file
- this function maps the THashFile signature


1.6.126. HtmlEscapeMarkdown

function HtmlEscapeMarkdown(const md: RawUtf8; esc: TTextWriterHtmlEscape = [heEmojiToUtf8]): RawUtf8;

Escape some Markdown-marked text into HTML
- just a wrapper around AddHtmlEscapeMarkdown() process


1.6.127. HtmlEscapeWiki

function HtmlEscapeWiki(const wiki: RawUtf8; esc: TTextWriterHtmlEscape = [heHtmlEscape, heEmojiToUtf8]): RawUtf8;

Escape some wiki-marked text into HTML
- just a wrapper around AddHtmlEscapeWiki() process


1.6.128. IncludeTrailingUriDelimiter

function IncludeTrailingUriDelimiter(const URI: RawByteString): RawByteString;

Ensure the supplied URI contains a trailing '/' charater


1.6.129. IsBase64

function IsBase64(const s: RawByteString): boolean; overload;

Check if the supplied text is a valid Base64 encoded stream


1.6.130. IsBase64

function IsBase64(sp: PAnsiChar; len: PtrInt): boolean; overload;

Check if the supplied text is a valid Base64 encoded stream


1.6.131. isBlobHex

function isBlobHex(P: PUtf8Char): boolean;

Return true if the TEXT is encoded as SQLite3 BLOB literals (X'53514C697465' e.g.)


1.6.132. IsContentCompressed

function IsContentCompressed(Content: pointer; Len: PtrInt): boolean;

Retrieve if some content is compressed, from a supplied binary buffer
- returns TRUE, if the header in binary buffer "may" be compressed (this method can trigger false positives), e.g. begin with most common already compressed zip/gz/gif/png/jpeg/avi/mp3/mp4 markers (aka "magic numbers")


1.6.133. IsContentTypeCompressible

function IsContentTypeCompressible(ContentType: PUtf8Char): boolean;

Recognize e.g. 'text/css' or 'application/json' as compressible


1.6.134. IsContentTypeCompressibleU

function IsContentTypeCompressibleU(const ContentType: RawUtf8): boolean;

Recognize e.g. 'text/css' or 'application/json' as compressible


1.6.135. IsContentTypeJson

function IsContentTypeJson(ContentType: PUtf8Char): boolean;

Recognize e.g. 'application/json' or 'application/vnd.api+json'


1.6.136. IsStreamBuffer

function IsStreamBuffer(S: TStream): boolean;

Check if class is a TCustomMemoryStream/TRawByteStringStream


1.6.137. IsUrlValid

function IsUrlValid(P: PUtf8Char): boolean;

Checks if the supplied UTF-8 text don't need URI encoding
- returns TRUE if all its chars are non-void plain ASCII-7 RFC compatible identifiers (0..9a..zA..Z-_.~)


1.6.138. LogEscape

function LogEscape(source: PAnsiChar; sourcelen: integer; var temp: TLogEscape; enabled: boolean = true): PAnsiChar;

Fill TLogEscape stack buffer with the (hexadecimal) chars of the input binary
- up to 512 bytes will be escaped and appended to a local temp: TLogEscape variable, using the EscapeBuffer() low-level function
- you can then log the resulting escaped text by passing the returned PAnsiChar as % parameter to a TSynLog.Log() method
- the "enabled" parameter can be assigned from a process option, avoiding to process the escape if verbose logs are disabled
- used e.g. to implement logBinaryFrameContent option for WebSockets


1.6.139. LogEscapeFull

function LogEscapeFull(const source: RawByteString): RawUtf8; overload;

Returns a text buffer with the (hexadecimal) chars of the input binary
- is much slower than LogEscape/EscapeToShort, but has no size limitation


1.6.140. LogEscapeFull

function LogEscapeFull(source: PAnsiChar; sourcelen: integer): RawUtf8; overload;

Returns a text buffer with the (hexadecimal) chars of the input binary
- is much slower than LogEscape/EscapeToShort, but has no size limitation


1.6.141. MultiPartFormDataAddField

function MultiPartFormDataAddField(const FieldName, FieldValue: RawUtf8; var MultiPart: TMultiPartDynArray; const ForcedContentType: RawUtf8 = ''): boolean;

Encode a field in a multipart array
- FieldName: field name of the part
- FieldValue: value of the field
- Multipart: where the part is added
- consider THttpMultiPartStream from mormot.net.client for huge file content


1.6.142. MultiPartFormDataAddFile

function MultiPartFormDataAddFile(const FileName: TFileName; var MultiPart: TMultiPartDynArray; const Name: RawUtf8 = ''; const ForcedContentType: RawUtf8 = ''): boolean;

Encode a file in a multipart array
- FileName: file to encode
- Multipart: where the part is added
- Name: name of the part, is empty the name 'File###' is generated
- consider THttpMultiPartStream from mormot.net.client for huge file content


1.6.143. MultiPartFormDataDecode

function MultiPartFormDataDecode(const MimeType, Body: RawUtf8; var MultiPart: TMultiPartDynArray): boolean;

Decode multipart/form-data POST request content into memory
- following RFC 1867
- decoded sections are appended to MultiPart[] existing array


1.6.144. MultiPartFormDataEncode

function MultiPartFormDataEncode(const MultiPart: TMultiPartDynArray; var MultiPartContentType, MultiPartContent: RawUtf8; Rfc2388NestedFiles: boolean = false): boolean;

Encode multipart fields and files
- only one of them can be used because MultiPartFormDataDecode must implement both decodings
- MultiPart: parts to build the multipart content from, which may be created using MultiPartFormDataAddFile/MultiPartFormDataAddField
- MultiPartContentType: variable returning

 Content-Type: multipart/form-data; boundary=xxx

where xxx is the first generated boundary
- MultiPartContent: generated multipart content
- Rfc2388NestedFiles will force the deprecated nested "multipart/mixed" format
- consider THttpMultiPartStream from mormot.net.client for huge file content


1.6.145. MultiPartFormDataNewBound

function MultiPartFormDataNewBound(var boundaries: TRawUtf8DynArray): RawUtf8;

Used e.g. by MultiPartFormDataEncode and THttpMultiPartStream.Add


1.6.146. Plural

function Plural(const itemname: ShortString; itemcount: cardinal): ShortString;

Write count number and append 's' (if needed) to form a plural English noun
- for instance, Plural('row',100) returns '100 rows' with no heap allocation


1.6.147. RawBlobToBlob

function RawBlobToBlob(const RawBlob: RawBlob): RawUtf8; overload;

Creates a TEXT-encoded version of blob data from a RawBlob
- TEXT will be encoded as SQLite3 BLOB literals (X'53514C697465' e.g.)


1.6.148. RawBlobToBlob

function RawBlobToBlob(RawBlob: pointer; RawBlobLength: integer): RawUtf8; overload;

Creates a TEXT-encoded version of blob data from a memory data
- same as RawBlob, but with direct memory access via a pointer/byte size pair
- TEXT will be encoded as SQLite3 BLOB literals (X'53514C697465' e.g.)


1.6.149. RawByteStringArrayConcat

function RawByteStringArrayConcat(const Values: array of RawByteString): RawByteString;

Fast concatenation of several AnsiStrings


1.6.150. RawByteStringToBytes

procedure RawByteStringToBytes(const buf: RawByteString; out bytes: TBytes);

Creates a TBytes from a RawByteString memory buffer


1.6.151. ResourceSynLZToRawByteString

procedure ResourceSynLZToRawByteString(const ResName: string; out buf: RawByteString; Instance: TLibHandle = 0);

Creates a RawByteString memory buffer from an SynLZ-compressed embedded resource
- returns '' if the resource is not found
- this method would use SynLZDecompress() after ResourceToRawByteString(), with a ResType=PChar(10) (i.e. RC_DATA)
- you can specify a library (dll) resource instance handle, if needed


1.6.152. ResourceToRawByteString

procedure ResourceToRawByteString(const ResName: string; ResType: PChar; out buf: RawByteString; Instance: TLibHandle = 0);

Creates a RawByteString memory buffer from an embedded resource
- returns '' if the resource is not found
- warning: resources size may be rounded up to alignment
- you can specify a library (dll) resource instance handle, if needed


1.6.153. SameFileContent

function SameFileContent(const One, Another: TFileName): boolean;

Compare two files by content, reading them by blocks


1.6.154. ToVarInt32

function ToVarInt32(Value: PtrInt; Dest: PByte): PByte;

Convert an integer into a 32-bit variable-length integer buffer
- store negative values as cardinal two-complement, i.e. 0=0,1=1,2=-1,3=2,4=-2...


1.6.155. ToVarInt64

function ToVarInt64(Value: Int64; Dest: PByte): PByte;

Convert a Int64 into a 64-bit variable-length integer buffer


1.6.156. ToVarString

function ToVarString(const Value: RawUtf8; Dest: PByte): PByte;

Convert a RawUtf8 into an UTF-8 encoded variable-length buffer


1.6.157. ToVarUInt32

function ToVarUInt32(Value: cardinal; Dest: PByte): PByte;

Convert a cardinal into a 32-bit variable-length integer buffer


1.6.158. ToVarUInt32Length

function ToVarUInt32Length(Value: PtrUInt): PtrUInt;

Return the number of bytes necessary to store a 32-bit variable-length integer
- i.e. the ToVarUInt32() buffer size


1.6.159. ToVarUInt32LengthWithData

function ToVarUInt32LengthWithData(Value: PtrUInt): PtrUInt;

Return the number of bytes necessary to store some data with a its 32-bit variable-length integer length


1.6.160. ToVarUInt64

function ToVarUInt64(Value: QWord; Dest: PByte): PByte;

Convert a UInt64 into a 64-bit variable-length integer buffer


1.6.161. UrlAppend

function UrlAppend(const Uri: RawUtf8; const NameValuePairs: array of const; Options: TUrlEncoder = []): RawUtf8;

Append some encoded Name,Value pairs parameters to an existing URI
- will check if Uri does already end with '?' or '&'


1.6.162. UrlDecode

function UrlDecode(const s: RawUtf8): RawUtf8; overload;

Decode a UrlEncode() URI encoded parameter into its original value


1.6.163. UrlDecode

function UrlDecode(U: PUtf8Char): RawUtf8; overload;

Decode a UrlEncode() URI encoded parameter into its original value


1.6.164. UrlDecodeCardinal

function UrlDecodeCardinal(U: PUtf8Char; const Upper: RawUtf8; var Value: cardinal; Next: PPUtf8Char = nil): boolean;

Decode a specified parameter compatible with URI encoding into its original cardinal numerical value
- UrlDecodeCardinal('offset=20&where=LastName%3D%27M%C3%B4net%27','OFFSET=',O,@Next) will return Next^='where=...' and O=20
- Upper should be already uppercased, and end with a '=' character
- if Upper is not found, Value is not modified, and result is FALSE
- if Upper is found, Value is modified with the supplied content, and result is TRUE


1.6.165. UrlDecodeDouble

function UrlDecodeDouble(U: PUtf8Char; const Upper: RawUtf8; var Value: double; Next: PPUtf8Char = nil): boolean;

Decode a specified parameter compatible with URI encoding into its original floating-point value
- UrlDecodeDouble('price=20.45&where=LastName%3D%27M%C3%B4net%27','PRICE=',P,@Next) will return Next^='where=...' and P=20.45
- Upper should be already uppercased, and end with a '=' character
- if Upper is not found, Value is not modified, and result is FALSE
- if Upper is found, Value is modified with the supplied content, and result is TRUE


1.6.166. UrlDecodeExtended

function UrlDecodeExtended(U: PUtf8Char; const Upper: RawUtf8; var Value: TSynExtended; Next: PPUtf8Char = nil): boolean;

Decode a specified parameter compatible with URI encoding into its original floating-point value
- UrlDecodeExtended('price=20.45&where=LastName%3D%27M%C3%B4net%27','PRICE=',P,@Next) will return Next^='where=...' and P=20.45
- Upper should be already uppercased, and end with a '=' character
- if Upper is not found, Value is not modified, and result is FALSE
- if Upper is found, Value is modified with the supplied content, and result is TRUE


1.6.167. UrlDecodeInt64

function UrlDecodeInt64(U: PUtf8Char; const Upper: RawUtf8; var Value: Int64; Next: PPUtf8Char = nil): boolean;

Decode a specified parameter compatible with URI encoding into its original Int64 numerical value
- UrlDecodeInt64('offset=20&where=LastName%3D%27M%C3%B4net%27','OFFSET=',O,@Next) will return Next^='where=...' and O=20
- Upper should be already uppercased, and end with a '=' character
- if Upper is not found, Value is not modified, and result is FALSE
- if Upper is found, Value is modified with the supplied content, and result is TRUE


1.6.168. UrlDecodeInteger

function UrlDecodeInteger(U: PUtf8Char; const Upper: RawUtf8; var Value: integer; Next: PPUtf8Char = nil): boolean;

Decode a specified parameter compatible with URI encoding into its original integer numerical value
- UrlDecodeInteger('offset=20&where=LastName%3D%27M%C3%B4net%27','OFFSET=',O,@Next) will return Next^='where=...' and O=20
- Upper should be already uppercased, and end with a '=' character
- if Upper is not found, Value is not modified, and result is FALSE
- if Upper is found, Value is modified with the supplied content, and result is TRUE


1.6.169. UrlDecodeName

function UrlDecodeName(U: PUtf8Char): RawUtf8; overload;

Decode a UrlEncodeName() URI encoded network name into its original value
- only parameters - i.e. after '?' - should replace spaces by '+'


1.6.170. UrlDecodeName

function UrlDecodeName(const s: RawUtf8): RawUtf8; overload;

Decode a UrlEncodeName() URI encoded network name into its original value
- only parameters - i.e. after '?' - should replace spaces by '+'


1.6.171. UrlDecodeNeedParameters

function UrlDecodeNeedParameters(U, CsvNames: PUtf8Char): boolean;

Returns TRUE if all supplied parameters do exist in the URI encoded text
- CsvNames parameter shall provide as a CSV list of names
- e.g. UrlDecodeNeedParameters('price=20.45&where=LastName%3D','price,where') will return TRUE


1.6.172. UrlDecodeNextName

function UrlDecodeNextName(U: PUtf8Char; out Name: RawUtf8): PUtf8Char;

Decode a URI-encoded Name from an input buffer
- decoded value is set in Name out variable
- returns a pointer just after the decoded name, after the '='
- returns nil if there was no name=... pattern in U


1.6.173. UrlDecodeNextNameValue

function UrlDecodeNextNameValue(U: PUtf8Char; var Name, Value: RawUtf8): PUtf8Char;

Decode the next Name=Value&.... pair from input URI
- Name is returned directly (should be plain ASCII 7-bit text)
- Value is returned after URI decoding (from %.. patterns)
- if a pair is decoded, return a PUtf8Char pointer to the next pair in the input buffer, or points to #0 if all content has been processed
- if a pair is not decoded, return nil


1.6.174. UrlDecodeNextValue

function UrlDecodeNextValue(U: PUtf8Char; out Value: RawUtf8): PUtf8Char;

Decode a URI-encoded Value from an input buffer
- decoded value is set in Value out variable
- returns a pointer just after the decoded value (may points e.g. to

0 or '&') - it is up to the caller to continue the process or not

1.6.175. UrlDecodeValue

function UrlDecodeValue(U: PUtf8Char; const Upper: RawUtf8; var Value: RawUtf8; Next: PPUtf8Char = nil): boolean;

Decode a specified parameter compatible with URI encoding into its original textual value
- UrlDecodeValue('select=%2A&where=LastName%3D%27M%C3%B4net%27','SELECT=',V,@Next) will return Next^='where=...' and V='*'
- Upper should be already uppercased, and end with a '=' character
- if Upper is not found, Value is not modified, and result is FALSE
- if Upper is found, Value is modified with the supplied content, and result is TRUE


1.6.176. UrlDecodeVar

procedure UrlDecodeVar(U: PUtf8Char; L: PtrInt; var result: RawUtf8; name: boolean);

Decode a UrlEncode/UrlEncodeName() URI encoded string into its original value
- name=false for parameters (after ?), to replace spaces by '+'


1.6.177. UrlEncode

function UrlEncode(Text: PUtf8Char): RawUtf8; overload;

Encode a string as URI parameter encoding, i.e. ' ' as '+'


1.6.178. UrlEncode

procedure UrlEncode(W: TTextWriter; Text: PUtf8Char; TextLen: PtrInt); overload;

Append a string as URI parameter encoding, i.e. ' ' as '+'


1.6.179. UrlEncode

procedure UrlEncode(W: TTextWriter; const Text: RawUtf8); overload;

Append a string as URI parameter encoding, i.e. ' ' as '+'


1.6.180. UrlEncode

function UrlEncode(const Text: RawUtf8): RawUtf8; overload;

Encode a string as URI parameter encoding, i.e. ' ' as '+'


1.6.181. UrlEncode

function UrlEncode(const NameValuePairs: array of const; TrimLeadingQuestionMark: boolean): RawUtf8; overload;

Encode supplied parameters to be compatible with URI encoding
- consider using UrlAppend() if you just need to append some parameters


1.6.182. UrlEncode

function UrlEncode(const NameValuePairs: array of const; Options: TUrlEncoder = []): RawUtf8; overload;

Encode supplied parameters to be compatible with URI encoding
- parameters must be supplied two by two, as Name,Value pairs, e.g.

 url := UrlEncode(['select','*','where','ID=12','offset',23,'object',aObject]);

- parameters names should be plain ASCII-7 RFC compatible identifiers (0..9a..zA..Z_.~), otherwise they are skipped unless ueEncodeNames is set
- parameters values can be either textual, integer or extended, or any TObject
- TObject serialization into UTF-8 will be processed with ObjectToJson()


1.6.183. UrlEncodeFull

function UrlEncodeFull(const PrefixFmt: RawUtf8; const PrefixArgs, NameValuePairs: array of const; Options: TUrlEncoder): RawUtf8;

Encode a full URI with prefix and parameters


1.6.184. UrlEncodeName

procedure UrlEncodeName(W: TTextWriter; Text: PUtf8Char; TextLen: PtrInt); overload;

Append a string as URI network name encoding, i.e. ' ' as %20
- only parameters - i.e. after '?' - should replace spaces by '+'


1.6.185. UrlEncodeName

procedure UrlEncodeName(W: TTextWriter; const Text: RawUtf8); overload;

Append a string as URI network name encoding, i.e. ' ' as %20
- only parameters - i.e. after '?' - should replace spaces by '+'


1.6.186. UrlEncodeName

function UrlEncodeName(const Text: RawUtf8): RawUtf8; overload;

Encode a string as URI network name encoding, i.e. ' ' as %20
- only parameters - i.e. after '?' - should replace spaces by '+'


1.6.187. UrlEncodeName

function UrlEncodeName(Text: PUtf8Char): RawUtf8; overload;

Encode a string as URI network name encoding, i.e. ' ' as %20
- only parameters - i.e. after '?' - should replace spaces by '+'


1.7. Variables implemented in the mormot.core.buffers unit

1.7.1. AlgoRle

AlgoRle: TAlgoCompress;

Run-Length-Encoding (RLE) compression as a TAlgoCompress class
- if RLE has no effect during compression, will fallback to plain store


1.7.2. AlgoRleLZ

AlgoRleLZ: TAlgoCompress;

SynLZ compression with RLE preprocess as a TAlgoCompress class
- SynLZ is not efficient when its input has a lot of identical characters (e.g. a database content, or a raw binary buffer) - try with this class which is slower than AlgoSynLZ but may have better ratio on such content


1.7.3. AlgoSynLZ

AlgoSynLZ: TAlgoCompress;

Our fast SynLZ compression as a TAlgoCompress class
- please use this global variable methods instead of the deprecated SynLZCompress/SynLZDecompress wrapper functions


1.7.4. Base64DecodeMain

Base64DecodeMain: function(sp, rp: PAnsiChar; len: PtrInt): boolean;

Raw function for efficient Base64 to binary decoding of the main block
- don't use this function, but rather the Base64ToBin() overloaded functions
- on FPC x86_64, detect and use AVX2 asm for very high throughput (9GB/s)


1.7.5. Base64EncodeMain

Base64EncodeMain: function(rp, sp: PAnsiChar; len: cardinal): integer;

Raw function for efficient binary to Base64 encoding of the main block
- don't use this function, but rather the BinToBase64() overloaded functions
- on FPC x86_64, detect and use AVX2 asm for very high throughput (11GB/s)


1.7.6. EMOJI_AFTERDOTS

EMOJI_AFTERDOTS: array['('..'|'] of TEmoji;

To recognize simple :) :( :| :/ :D :o :p :s characters as smilleys


1.7.7. EMOJI_RTTI

EMOJI_RTTI: PShortString;

Low-level access to TEmoji RTTI - used when inlining EmojiFromText()


1.7.8. EMOJI_TAG

EMOJI_TAG: array[TEmoji] of RawUtf8;

Github/Markdown compatible tag of Emojis, including trailing and ending :
- e.g. ':grinning:' or ':person_with_pouting_face:'


1.7.9. EMOJI_TEXT

EMOJI_TEXT: array[TEmoji] of RawUtf8;

Github/Markdown compatible text of Emojis
- e.g. 'grinning' or 'person_with_pouting_face'


1.7.10. EMOJI_UTF8

EMOJI_UTF8: array[TEmoji] of RawUtf8;

The Unicode character matching a given Emoji, after UTF-8 encoding


1.7.11. LogCompressAlgo

LogCompressAlgo: TAlgoCompress;

Define how files are compressed by TSynLog.PerformRotation
- as used within mormot.core.log.pas unit, and defined in this unit to be available wihout any dependency to it (e.g. in compression units)
- assigned to AlgoSynLZ by default for .synlz which is the fastest for logs
- you may set AlgoGZFast from mormot.core.zip.pas to generate .gz standard files during TSynLog file rotation (with libdeflate if available)
- you may set AlgoLizardFast or AlgoLizardHuffman as non-standard alternatives (default AlgoLizard is much slower and less efficient on logs)
- if you set nil, no compression will take place during rotation
- note that compression itself is run in the logging background thread


1.7.12. LogCompressAlgoArchive

LogCompressAlgoArchive: function(aAlgo: TAlgoCompress; aMagic: cardinal; const aOldLogFileName, aDestinationPath: TFileName): boolean;

Internal wrapper function used by TSynLogArchiveEvent handlers to compress and delete older .log files using our proprietary FileCompress format for a given algorithm
- as used within mormot.core.log.pas unit, and defined in this unit to be available wihout any dependency to it (e.g. in compression units)
- called by EventArchiveLizard/EventArchiveSynLZ to implement .synlz/.synliz archival