Purpose: Framework Core Low-Level Memory Buffer Process
- this unit is a part of the Open Source Synopse mORMot framework 2, licensed under a MPL/GPL/LGPL three license - see LICENSE.md
Unit Name | Description | |
---|---|---|
mormot.core.base | Framework Core Shared Types and RTL-like Functions | |
mormot.core.os | Framework Core Low-Level Wrappers to the Operating-System API | |
mormot.core.rtti | Framework Core Low-Level Cross-Compiler RTTI Definitions | |
mormot.core.text | Framework Core Low-Level Text Processing | |
mormot.core.unicode | Framework Core Low-Level Unicode UTF-8 UTF-16 Ansi Conversion |
Objects | Description | |
---|---|---|
EAlgoCompress | Exception raised by TAlgoCompress classes | |
EBufferException | Exception raised during buffer processing | |
EFastReader | Exception raised during TFastReader decoding | |
EStreamRedirect | Exception raised during TStreamRedirect processing | |
TAlgoCompress | Abstract low-level parent class for generic compression/decompression algorithms | |
TAlgoCompressWithNoDestLen | Abstract class storing the plain length before calling compression API | |
TAlgoRle | Implement RLE compression as a TAlgoCompress class | |
TAlgoRleLZ | Implement our SynLZ compression with RLE preprocess as a TAlgoCompress class | |
TAlgoSynLZ | Implement our fast SynLZ compression as a TAlgoCompress class | |
TBufferedStreamReader | TStream with an internal memory buffer | |
TBufferWriter | This class can be used to speed up writing to a file or a stream | |
TFakeWriterStream | A fake TStream, which will just count the number of bytes written | |
TFastReader | Safe decoding of a TBufferWriter content from an in-memory buffer | |
TMemoryMapText | Able to read a UTF-8 text file using memory map | |
TMultiPart | Used by MultiPartFormDataDecode() to return one item of its data | |
TNestedStreamReader | TStream allowing to read from some nested TStream instances | |
TProgressInfo | Information about the progression of a process, e.g. for TStreamRedirect | |
TRawByteStringBuffer | Store one RawByteString content with an associated length | |
TRawByteStringGroup | Store several RawByteString content with optional concatenation | |
TRawByteStringGroupValue | Item as stored in a TRawByteStringGroup instance | |
TRawByteStringHeap | Thread-safe reusable set of constant RawByteString/RawUtf8 instances | |
TStreamRedirect | An abstract pipeline stream able to redirect and hash read/written content | |
TStreamRedirectCrc32c | TStreamRedirect with crc32c 32-bit checksum | |
TStreamRedirectHash32 | TStreamRedirect with 32-bit THasher checksum | |
TValueResult | Kind of result returned by FromVarBlob() function |
TValueResult = record
Kind of result returned by FromVarBlob
() function
Len: PtrInt;
Value length (in bytes)
Ptr: PAnsiChar;
Start of data value
EAlgoCompress = class(ESynException)
Exception raised by TAlgoCompress
classes
TAlgoCompress = class(TObject)
Abstract low-level parent class for generic compression/decompression algorithms
- will encapsulate the compression algorithm with crc32c
hashing
- all Algo* abstract methods should be overriden by inherited classes
- don't inherit from TSynPersistent
since we don't need any of it
constructor Create; virtual;
Will register AlgoID
in the global list, for Algo
() class methods
- no need to free this instance, since it will be owned by the global list
- raise a EAlgoCompress
if the class or its AlgoID
are already registered
- you should never have to call this constructor, but define a global variable holding a reference to a shared instance
destructor Destroy; override;
Finalize this algorithm
class function Algo(Comp: PAnsiChar; CompLen: integer): TAlgoCompress; overload;
Get the TAlgoCompress
instance corresponding to the AlgoID
stored in the supplied compressed buffer
- returns nil if no algorithm was identified
class function Algo(const Comp: RawByteString): TAlgoCompress; overload;
Get the TAlgoCompress
instance corresponding to the AlgoID
stored in the supplied compressed buffer
- returns nil if no algorithm was identified
class function Algo(const Comp: TByteDynArray): TAlgoCompress; overload;
Get the TAlgoCompress
instance corresponding to the AlgoID
stored in the supplied compressed buffer
- returns nil if no algorithm was identified
class function Algo(aAlgoID: byte): TAlgoCompress; overload;
Get the TAlgoCompress
instance corresponding to the supplied AlgoID
- returns nil if no algorithm was identified
- stored content is identified as TAlgoSynLZ
class function Algo(Comp: PAnsiChar; CompLen: integer; out IsStored: boolean): TAlgoCompress; overload;
Get the TAlgoCompress
instance corresponding to the AlgoID
stored in the supplied compressed buffer
- returns nil if no algorithm was identified
- also identifies "stored" content in IsStored variable
function AlgoCompress(Plain: pointer; PlainLen: integer; Comp: pointer): integer; virtual; abstract;
This method will compress
the supplied data
function AlgoCompressDestLen(PlainLen: integer): integer; virtual; abstract;
Get maximum possible (worse) compressed size for the supplied length
function AlgoDecompress(Comp: pointer; CompLen: integer; Plain: pointer): integer; virtual; abstract;
This method will decompress
the supplied data
function AlgoDecompressDestLen(Comp: pointer): integer; virtual; abstract;
This method will return the size of the decompressed data
function AlgoDecompressPartial(Comp: pointer; CompLen: integer; Partial: pointer; PartialLen, PartialLenMax: integer): integer; virtual; abstract;
This method will partially and safely decompress
the supplied data
- expects PartialLen <= result < PartialLenMax, depending on the algorithm
function AlgoHash(ForceHash32: boolean; Data: pointer; DataLen: integer): cardinal; overload;
Computes the digital signature of the buffer, or Hash32
() if specified
function AlgoHash(Previous: cardinal; Data: pointer; DataLen: integer): cardinal; overload; virtual;
Computes by default the crc32c
() digital signature of the buffer
function AlgoName: TShort16;
Returns the algorithm name, from its classname
- e.g. TAlgoSynLZ
->'synlz' TAlgoLizard->'lizard
' nil->'none' TAlgoDeflateFast->'deflatefast'
function Compress(Plain: PAnsiChar; PlainLen: integer; CompressionSizeTrigger: integer = 100; CheckMagicForCompressed: boolean = false; BufferOffset: integer = 0): RawByteString; overload; virtual;
Compress
a memory buffer with crc32c
hashing to a RawByteString
function Compress(const Plain: RawByteString; CompressionSizeTrigger: integer = 100; CheckMagicForCompressed: boolean = false; BufferOffset: integer = 0): RawByteString; overload;
Compress
a memory buffer with crc32c
hashing to a RawByteString
function Compress(Plain, Comp: PAnsiChar; PlainLen, CompLen: integer; CompressionSizeTrigger: integer = 100; CheckMagicForCompressed: boolean = false): integer; overload; virtual;
Compress
a memory buffer with crc32c
hashing
- supplied Comp buffer should contain at least CompressDestLen
(PlainLen) bytes
function CompressDestLen(PlainLen: integer): integer;
Get maximum possible (worse) compressed size for the supplied length
- including the crc32c
+ algo
9 bytes header
function CompressToBytes(Plain: PAnsiChar; PlainLen: integer; CompressionSizeTrigger: integer = 100; CheckMagicForCompressed: boolean = false): TByteDynArray; overload;
Compress
a memory buffer with crc32c
hashing to a TByteDynArray
function CompressToBytes(const Plain: RawByteString; CompressionSizeTrigger: integer = 100; CheckMagicForCompressed: boolean = false): TByteDynArray; overload;
Compress
a memory buffer with crc32c
hashing to a TByteDynArray
function Decompress(const Comp: TByteDynArray): RawByteString; overload;
Uncompress a RawByteString
memory buffer with crc32c
hashing
function Decompress(const Comp: RawByteString; out PlainLen: integer; var tmp: RawByteString; Load: TAlgoCompressLoad = aclNormal): pointer; overload;
Uncompress a RawByteString
memory buffer with crc32c
hashing
- returns nil if crc32
hash
failed, i.e. if the supplied Comp is not correct
- returns a pointer to the uncompressed data and fill PlainLen variable, after crc32c
hash
- avoid any memory allocation in case of a stored content - otherwise, would uncompress to the tmp variable, and return pointer(tmp) and length(tmp)
function Decompress(Comp: PAnsiChar; CompLen: integer; out PlainLen: integer; var tmp: RawByteString; Load: TAlgoCompressLoad = aclNormal): pointer; overload;
Uncompress a RawByteString
memory buffer with crc32c
hashing
- returns nil if crc32
hash
failed, i.e. if the supplied Data is not correct
- returns a pointer to an uncompressed data buffer of PlainLen bytes
- avoid any memory allocation in case of a stored content - otherwise, would uncompress to the tmp variable, and return pointer(tmp) and length(tmp)
function Decompress(const Comp: RawByteString; Load: TAlgoCompressLoad = aclNormal; BufferOffset: integer = 0): RawByteString; overload;
Uncompress a RawByteString
memory buffer with crc32c
hashing
function DecompressBody(Comp, Plain: PAnsiChar; CompLen, PlainLen: integer; Load: TAlgoCompressLoad = aclNormal): boolean; virtual;
Decode the content of a memory buffer compressed via the Compress
() method
- PlainLen has been returned by a previous call to DecompressHeader
()
function DecompressHeader(Comp: PAnsiChar; CompLen: integer; Load: TAlgoCompressLoad = aclNormal): integer; virtual;
Decode the header of a memory buffer compressed via the Compress
() method
- validates the crc32c
of the compressed data (unless Load=aclNoCrcFast), then return the uncompressed size in bytes, or 0 if the crc32c
does not match
- should call DecompressBody
() later on to actually retrieve the content
function DecompressPartial(Comp, Partial: PAnsiChar; CompLen, PartialLen, PartialLenMax: integer): integer; virtual;
Partial decoding of a memory buffer compressed via the Compress
() method
- returns 0 on error, or how many bytes have been written to Partial
- will call virtual AlgoDecompressPartial
() which is slower, but expected to avoid any buffer overflow on the Partial destination buffer
- some algorithms (e.g. Lizard
) may need some additional bytes in the decode buffer, so PartialLenMax bytes should be allocated in Partial^, with PartialLenMax > expected PartialLen, and returned bytes may be > PartialLen, but always <= PartialLenMax
function FileCompress(const Source, Dest: TFileName; Magic: cardinal; ForceHash32: boolean = false; ChunkBytes: Int64 = 128 shl 20; WithTrailer: boolean = false): boolean; virtual;
Compress
a file content using this compression algorithm
- source file is split
into ChunkBytes blocks (128 MB by default) for fast in-memory compression of any file size, then compressed and including checksums of Source/Dest data
- it is not compatible with StreamCompress
format, which has no chunking
- you should specify a Magic number to be used to identify the compressed file format
- follow the FileSynLZ
() deprecated function format, if ForceHash32=true so that Hash32
() is used instead of the AlgoHash
() of this instance
- may be overriden to support a standard file layout (e.g. AlgoGZ
)
class function FileIsCompressed(const Name: TFileName; Magic: cardinal): boolean; virtual;
Returns TRUE if the supplied file name is a compressed file, matching the Magic number as supplied to FileCompress
() function
- follow the FileIsSynLZ
() deprecated function format
- expects the compressed data to be at file beginning (not appended)
- may be overriden to support a standard file layout (e.g. AlgoGZ
)
function FileUnCompress(const Source, Dest: TFileName; Magic: cardinal; ForceHash32: boolean = false): boolean; virtual;
Uncompress a file previously compressed via FileCompress
()
- you should specify a Magic number to be used to identify the compressed file format
- follow the FileUnSynLZ
() deprecated function format, if ForceHash32=true so that Hash32
() is used instead of the AlgoHash
() of this instance
- may be overriden to support a standard file layout (e.g. AlgoGZ
)
function StreamCompress(Source: TStream; const DestFile: TFileName; Magic: cardinal; ForceHash32: boolean = false; WithTrailer: boolean = true; ChunkBytes: PtrInt = 4 shl 20): Int64; overload;
Compress
a Stream content using this compression algorithm into a file
- just a wrapper around the overloaded StreamCompress
() method
function StreamCompress(Source, Dest: TStream; Magic: cardinal; ForceHash32: boolean = false; WithTrailer: boolean = true; ChunkBytes: PtrInt = 4 shl 20): Int64; overload;
Compress
a Stream content using this compression algorithm
- source Stream may be read and compressed by ChunkBytes = 4MB chunks
- a 32-bit Magic number identifies the compressed content chunks
- WithTrailer would finish the Dest output with a trailer block to locate the position of the compressed data, to be used e.g. when it is appended
- follow the StreamSynLZ
() deprecated function format, if ForceHash32=true and WithTrailer=true so that Hash32
() is used instead of AlgoHash
()
function StreamComputeLen(P: PAnsiChar; Len: PtrUInt; Magic: cardinal): integer;
Compute the length of a given StreamCompress
() buffer from its trailer
- allows to replace an existing appended content, for instance
- expects StreamCompress
(WithTrailer=true) format
function StreamUnCompress(Source, Dest: TStream; Magic: cardinal; ForceHash32: boolean = false): boolean; overload;
Uncompress a Stream previously compressed via StreamCompress
()
- you should specify a Magic number to be used to identify the compressed Stream format
- if Source is not positioned at compressed data beginning, a trailer is searched at the end of the Source stream to get the proper location
- returns true on success, and false on decoding error - but some chunks may have been decompressed in Dest even if false is returned
function StreamUnCompress(const Source: TFileName; Magic: cardinal; ForceHash32: boolean = false): TMemoryStream; overload;
Uncompress a File previously compressed via StreamCompress
() as TStream
- you should specify a Magic number to be used to identify the compressed Stream format
- follow the StreamUnSynLZ
() deprecated function format, if ForceHash32=true so that Hash32
() is used instead of the AlgoHash
() of this instance
- if the compressed data is not at Source file beginning, a trailer is searched at the end of the Source content to get the proper location
function StreamUnCompress(Source: TStream; Magic: cardinal; ForceHash32: boolean = false): TMemoryStream; overload;
Uncompress a Stream previously compressed via StreamCompress
()
- return nil on decompression error, or a new TMemoryStream instance
- follow the StreamUnSynLZ
() deprecated function format, if ForceHash32=true so that Hash32
() is used instead of the AlgoHash
() of this instance
function TryDecompress(const Comp: RawByteString; out Dest: RawByteString; Load: TAlgoCompressLoad = aclNormal): boolean;
Uncompress a RawByteString
memory buffer with crc32c
hashing
- returns TRUE on success
class function UncompressedSize(const Comp: RawByteString): integer;
Quickly validate a compressed buffer content, without uncompression
- extract the TAlgoCompress
, and call DecompressHeader
() to check the hash
of the compressed data, and return then uncompressed size
- returns 0 on error (e.g. unknown algorithm or incorrect hash
)
procedure Decompress(Comp: PAnsiChar; CompLen: integer; out result: RawByteString; Load: TAlgoCompressLoad = aclNormal; BufferOffset: integer = 0); overload;
Uncompress a memory buffer with crc32c
hashing
property AlgoFileExt: TFileName read fAlgoFileExt;
The usual file extension of this algorithm
- e.g. '.synlz' or '.synz' or '.synliz' for SynLZ, Deflate or Lizard
property AlgoHasForcedFormat: boolean read fAlgoHasForcedFormat;
If this algorithm does not supports our custom storage format
- e.g. AlgoGZ
set true and only supports plain buffers and files methods and would raise EAlgoCompress
when stream methods are used
property AlgoID: byte read fAlgoID;
Contains a genuine byte identifier for this algorithm
- 0 is reserved for stored, 1 for TAlgoSynLz
, 2/3 for TAlgoDeflate/Fast (in mormot.core.zip.pas), 4/5/6 for TAlgoLizard/Fast/Huffman (in mormot.lib.lizard.pas), 7/8 for TAlgoRleLZ
/TAlgoRle
, 9/10 for limited TAlgoGZ/TAlgoGZFast (in mormot.core.zip.pas)
TAlgoSynLZ = class(TAlgoCompress)
Implement our fast SynLZ compression as a TAlgoCompress
class
- please use the AlgoSynLZ
global variable methods instead of the deprecated SynLZCompress
/SynLZDecompress
wrapper functions
constructor Create; override;
Set AlgoID = 1 as genuine byte identifier for SynLZ
function AlgoCompress(Plain: pointer; PlainLen: integer; Comp: pointer): integer; override;
Compress the supplied data using SynLZ
function AlgoCompressDestLen(PlainLen: integer): integer; override;
Get maximum possible (worse) SynLZ compressed size for the supplied length
function AlgoDecompress(Comp: pointer; CompLen: integer; Plain: pointer): integer; override;
Decompress the supplied data using SynLZ
function AlgoDecompressDestLen(Comp: pointer): integer; override;
Return the size of the SynLZ decompressed data
function AlgoDecompressPartial(Comp: pointer; CompLen: integer; Partial: pointer; PartialLen, PartialLenMax: integer): integer; override;
Partial (and safe) decompression of the supplied data using SynLZ
TAlgoCompressWithNoDestLen = class(TAlgoCompress)
Abstract class storing the plain length before calling compression API
- some libraries (e.g. Deflate or Lizard
) don't provide the uncompressed length from its output buffer - inherit from this class to store
this value as ToVarUInt32
, and override the RawProcess abstract protected method
function AlgoCompress(Plain: pointer; PlainLen: integer; Comp: pointer): integer; override;
Performs the compression, storing PlainLen and calling protected RawProcess
function AlgoDecompress(Comp: pointer; CompLen: integer; Plain: pointer): integer; override;
Performs the decompression, retrieving PlainLen and calling protected RawProcess
function AlgoDecompressDestLen(Comp: pointer): integer; override;
Return the size of the decompressed data (using FromVarUInt32
)
function AlgoDecompressPartial(Comp: pointer; CompLen: integer; Partial: pointer; PartialLen, PartialLenMax: integer): integer; override;
Performs the decompression, retrieving PlainLen and calling protected RawProcess
TAlgoRleLZ = class(TAlgoCompressWithNoDestLen)
Implement our SynLZ compression with RLE preprocess as a TAlgoCompress
class
- SynLZ is very good with JSON or text
, but not so efficient when its input has a lot of padding (e.g. a database file, or unoptimized raw binary)
- this class would make
a first pass with RleCompress
() before SynLZ
- if RLE has no effect during compression, will fallback to plain SynLZ with no RLE pass during decompression
constructor Create; override;
Set AlgoID = 7 as genuine byte identifier for RLE + SynLZ
function AlgoCompressDestLen(PlainLen: integer): integer; override;
Get maximum possible (worse) compressed size for the supplied length
TAlgoRle = class(TAlgoCompressWithNoDestLen)
Implement RLE compression as a TAlgoCompress
class
- if RLE has no effect during compression, will fallback to plain store
constructor Create; override;
Set AlgoID = 8 as genuine byte identifier for RLE
function AlgoCompressDestLen(PlainLen: integer): integer; override;
Get maximum possible (worse) compressed size for the supplied length
EFastReader = class(ESynException)
Exception raised during TFastReader
decoding
TFastReader = object(TObject)
Safe decoding of a TBufferWriter
content from an in-memory buffer
- raise a EFastReader
exception on decoding error (e.g. if a buffer overflow may occur) or call OnErrorOverflow/OnErrorData event handlers
CustomVariants: pointer;
When used to unserialize variants, stores options for TDocVariant
creation
- contains a PDocVariantOptions
reference pointer as defined in the mormot.core.data unit
Last: PAnsiChar;
The last
position in the buffer
OnErrorData: TOnFastReaderErrorData;
Use this event to customize the ErrorData
process
OnErrorOverflow: TOnFastReaderErrorOverflow;
Use this event to customize the ErrorOverflow
process
P: PAnsiChar;
The current position in the memory
Tag: PtrInt;
Some opaque value, e.g. a version number to define the binary layout
function CopySafe(Dest: pointer; DataLen: PtrInt): boolean;
Copy data from the current position, and move ahead the specified bytes
- this version won't call ErrorOverflow
, but return false on error
- returns true on read success
function EOF: boolean;
Returns TRUE if the current position is the end of the input stream
function Next(DataLen: PtrInt): pointer;
Returns the current position, and move ahead the specified bytes
function Next2: cardinal;
Read the next
2 bytes from the buffer as a 16-bit unsigned value
function Next2BigEndian: cardinal;
Read the next
2 bytes from the buffer as a 16-bit big-endian value
function Next4: cardinal;
Read the next
4 bytes from the buffer as a 32-bit unsigned value
function Next8: Qword;
Read the next
8 bytes from the buffer as a 64-bit unsigned value
function NextByte: byte;
Read the next
byte from the buffer
function NextByteEquals(Value: byte): boolean;
Consumes the next
byte from the buffer, if matches a given value
function NextByteSafe(dest: pointer): boolean;
Read the next
byte from the buffer, checking
function NextSafe(out Data: pointer; DataLen: PtrInt): boolean;
Returns the current position, and move ahead the specified bytes
function PeekVarInt32(out value: PtrInt): boolean;
Try to read the next
32-bit signed value from the buffer
- don't change the current position
function PeekVarUInt32(out value: PtrUInt): boolean;
Try to read the next
32-bit unsigned value from the buffer
- don't change the current position
function ReadCompressed(Load: TAlgoCompressLoad = aclNormal; BufferOffset: integer = 0): RawByteString;
Retrieve some TAlgoCompress
buffer, appended via Write()
- BufferOffset could be set to reserve some bytes before the uncompressed buffer
function ReadVarRawUtf8DynArray(var Values: TRawUtf8DynArray): PtrInt;
Retrieved RawUtf8
values encoded with TFileBufferWriter.WriteRawUtf8DynArray
- returns the number of items read into Values[] (may differ from length(Values))
function ReadVarUInt32Array(var Values: TIntegerDynArray): PtrInt;
Retrieved cardinal values encoded with TBufferWriter.WriteVarUInt32Array
- Values[] will be resized only if it is not long enough, to spare heap
- returns decoded count in Values[], which may not be length(Values)
- wkFakeMarker will return -count and the caller should make
the decoding
function ReadVarUInt64Array(var Values: TInt64DynArray): PtrInt;
Retrieved Int64 values encoded with TBufferWriter.WriteVarUInt64DynArray
- Values[] will be resized only if it is not long enough, to spare heap
- returns decoded count in Values[], which may not be length(Values)
function RemainingLength: PtrUInt;
Returns remaining length (difference between Last
and P)
function VarBlob: TValueResult; overload;
Read the next
pointer and length value from the buffer
function VarBlobSafe(out Value: TValueResult): boolean;
Read the next
pointer and length value from the buffer
- this version won't call ErrorOverflow
, but return false on error
- returns true on read success
function VarInt32: integer;
Read the next
32-bit signed value from the buffer
function VarInt64: Int64;
Read the next
64-bit signed value from the buffer
function VarShortString: ShortString;
Read the next
ShortString value from the buffer
function VarString: RawByteString; overload;
Read the next
RawByteString
value from the buffer
function VarString(CodePage: integer): RawByteString; overload;
Read the next
RawByteString
value from the buffer
function VarUInt32: cardinal;
Read the next
32-bit unsigned value from the buffer
function VarUInt32Safe(out Value: cardinal): boolean;
Read the next
32-bit unsigned value from the buffer
- this version won't call ErrorOverflow
, but return false on error
- returns true on read success
function VarUInt64: QWord;
Read the next
64-bit unsigned value from the buffer
function VarUtf8: RawUtf8; overload;
Read the next
RawUtf8
value from the buffer
function VarUtf8Safe(out Value: RawUtf8): boolean;
Read the next
RawUtf8
value from the buffer
- this version won't call ErrorOverflow
, but return false on error
- returns true on read success
procedure Copy(Dest: pointer; DataLen: PtrInt);
Copy data from the current position, and move ahead the specified bytes
procedure ErrorData(const msg: shortstring); overload;
Raise a EFastReader
with "Incorrect Data: ...." error message
procedure ErrorData(const fmt: RawUtf8; const args: array of const); overload;
Raise a EFastReader
with "Incorrect Data: ...." error message
procedure ErrorOverflow;
Raise a EFastReader
with "Reached End of Input" error message
procedure Init(Buffer: pointer; Len: PtrInt); overload;
Initialize the reader from a memory block
procedure Init(const Buffer: RawByteString); overload;
Initialize the reader from a RawByteString
content
procedure VarBlob(out result: TValueResult); overload;
Read the next
pointer and length value from the buffer
procedure VarBlob(out Value: TSynTempBuffer); overload;
Copy the next
VarBlob
value from the buffer into a TSynTempBuffer
procedure VarNextInt(count: integer); overload;
Fast ignore the next
count VarUInt32
/VarInt32
/VarUInt64
/VarInt64
values
- don't raise any exception, so caller could check explicitly for any EOF
procedure VarNextInt; overload;
Fast ignore the next
VarUInt32
/VarInt32
/VarUInt64
/VarInt64
value
- don't raise any exception, so caller could check explicitly for any EOF
procedure VarUtf8(out result: RawUtf8); overload;
Read the next
RawUtf8
value from the buffer
EBufferException = class(ESynException)
Exception raised during buffer processing
TBufferWriter = class(TObject)
This class can be used to speed up writing to a file or a stream
- big speed up if data is written in small blocks
- also handle optimized storage of any integer/Int64/RawUtf8
values
- use TFileBufferReader or TFastReader
for decoding of the stored binary
constructor Create(aClass: TStreamClass; BufLen: integer = 4096); overload;
Initialize the buffer, using an owned TStream instance
- parameter could be e.g. TMemoryStream or TRawByteStringStream
- use Flush
then TMemoryStream(Stream
) to retrieve its content, or FlushTo
if TRawByteStringStream
was used
- Write() fails over 800MB (_STRMAXSIZE
) for a TRawByteStringStream
constructor Create(aClass: TStreamClass; aTempBuf: pointer; aTempLen: integer); overload;
Initialize with a specified buffer and an owned TStream
- use a specified external buffer (which may be allocated on stack), to avoid a memory allocation
- aClass could be e.g. TMemoryStream or TRawByteStringStream
constructor Create(const aStackBuffer: TTextWriterStackBuffer); overload;
Initialize with a stack-allocated 8KB of buffer
- destination stream
is an owned TRawByteStringStream
- so you can call FlushTo
to retrieve all written data
- Write() fails over 800MB (_STRMAXSIZE
) for a TRawByteStringStream
- convenient to reduce heap presure, when writing a few KB of data
constructor Create(aStream: TStream; aTempBuf: pointer; aTempLen: integer); overload;
Initialize with a specified buffer and an existing TStream instance
- use a specified external buffer (which may be allocated on stack), to avoid a memory allocation
constructor Create(aFile: THandle; BufLen: integer = 65536); overload;
Initialize the buffer, and specify a file handle to use for writing
- define an internal buffer of the specified size
constructor Create(aStream: TStream; BufLen: integer = 65536); overload;
Initialize the buffer, and specify a TStream to use for writing
- define an internal buffer of the specified size
constructor Create(const aFileName: TFileName; BufLen: integer = 65536; Append: boolean = false); overload;
Initialize the buffer, and specify a file to use for writing
- define an internal buffer of the specified size
- would replace any existing file by default, unless Append
is TRUE
destructor Destroy; override;
Release internal TStream (after AssignToHandle call)
- warning: an explicit call to Flush
is needed to write the data pending in internal buffer
function DirectWritePrepare(maxlen: PtrInt; var tmp: RawByteString): PAnsiChar;
Allows to write directly to a memory buffer
- caller should specify the maximum possible number of bytes to be written
- then write the data to the returned pointer, and call DirectWriteFlush
- if len is bigger than the internal buffer, tmp will be used instead
function DirectWriteReserve(maxlen: PtrInt): PByte;
Allows to write directly to a memory buffer
- caller should specify the maximum possible number of bytes to be written
- len should be smaller than the internal buffer size (not checked)
function Flush: Int64;
Write any pending data in the internal buffer to the stream
- after a Flush
, it's possible to call FileSeek64
(aFile,....)
- returns the number of bytes written between two FLush
method calls
function FlushAndCompress(nocompression: boolean = false; algo: TAlgoCompress = nil; BufferOffset: integer = 0): RawByteString;
Write any pending data, then call algo.Compress() on the buffer
- if algo is left to its default nil, will use global AlgoSynLZ
- features direct compression from internal buffer, if stream
was not used
- BufferOffset could be set to reserve some bytes before the compressed buffer
- raise an exception if internal Stream
is not a TRawByteStringStream
function FlushTo: RawByteString;
Write any pending data, then create
a RawByteString
from the content
- raise an exception if internal Stream
is not a TRawByteStringStream
function FlushToBytes: TBytes;
Write any pending data, then create
a TBytes
array from the content
- raise an exception if the size exceeds 800MB (_DAMAXSIZE
)
procedure CancelAll; virtual;
Rewind the Stream
to the position when Create
() was called
- note that this does not clear the Stream
content itself, just move back its writing position to its initial place
procedure DirectWriteFlush(len: PtrInt; const tmp: RawByteString);
Finalize a direct write to a memory buffer
- by specifying the number of bytes written to the buffer
procedure DirectWriteReserved(pos: PByte);
Flush
DirectWriteReserve
() content
procedure Write(const Text: RawByteString); overload;
Append
some length-prefixed UTF-8 text
at the current position
- will write the string length (as VarUInt32), then the string content, as expected by the FromVarString
() function
- is just a wrapper around WriteVar
()
procedure Write(Data: pointer; DataLen: PtrInt); overload;
Append
some data at the current position
- will be inlined as a MoveFast
() most of the time
procedure Write1(Data: byte);
Append
1 byte of data at the current position
procedure Write2(Data: cardinal);
Append
2 bytes of data at the current position
procedure Write2BigEndian(Data: cardinal);
Append
2 bytes of data, encoded as BigEndian, at the current position
procedure Write4(Data: integer);
Append
4 bytes of data at the current position
procedure Write4BigEndian(Data: integer);
Append
4 bytes of data, encoded as BigEndian, at the current position
procedure Write8(Data8Bytes: pointer);
Append
8 bytes of data at the current position
procedure WriteBinary(const Data: RawByteString);
Append
some content at the current position
- will write the binary data, without any length prefix
procedure WriteI64(Data: Int64);
Append
8 bytes of 64-bit integer at the current position
procedure WriteN(Data: byte; Count: integer);
Append
the same byte a given number of occurrences at the current position
procedure WriteRawUtf8Array(Values: PPtrUIntArray; ValuesCount: integer);
Append
a RawUtf8
array of values, from its low-level memory pointer
- handled the fixed size strings array case in a very efficient way
procedure WriteRawUtf8DynArray(const Values: TRawUtf8DynArray; ValuesCount: integer);
Append
the RawUtf8
dynamic array
- handled the fixed size strings array case in a very efficient way
procedure WriteShort(const Text: ShortString);
Append
some UTF-8 encoded text
at the current position
- will write the string length (as VarUInt32), then the string content
- is just a wrapper around WriteVar
()
procedure WriteStream(aStream: TCustomMemoryStream; aStreamSize: integer = -1);
Append
a TStream content
- is StreamSize is left as -1, the Stream.Size is used
- the size of the content is stored in the resulting stream
procedure WriteVar(var Item: TTempUtf8); overload;
Append
some TTempUtf8
text
content prefixed by its encoded length
- will also release any memory stored in Item.TempRawUtf8
procedure WriteVar(Data: pointer; DataLen: PtrInt); overload;
Append
some content (may be text
or binary) prefixed by its encoded length
- will write DataLen as VarUInt32, then the Data content, as expected by FromVarString
/FromVarBlob
functions
procedure WriteVarInt32(Value: PtrInt);
Append
an integer value using 32-bit variable-length integer encoding of the by-two complement of the given value
procedure WriteVarInt64(Value: Int64);
Append
an integer value using 64-bit variable-length integer encoding of the by-two complement of the given value
procedure WriteVarUInt32(Value: PtrUInt);
Append
a cardinal value using 32-bit variable-length integer encoding
procedure WriteVarUInt32Array(const Values: TIntegerDynArray; ValuesCount: integer; DataLayout: TBufferWriterKind);
Append
cardinal values (NONE must be negative!) using 32-bit variable-length integer encoding or other specialized algorithm, depending on the data layout
- could be decoded later on via TFastReader.ReadVarUInt32Array
procedure WriteVarUInt32Values(Values: PIntegerArray; ValuesCount: integer; DataLayout: TBufferWriterKind);
Append
cardinal values (NONE must be negative!) using 32-bit variable-length integer encoding or other specialized algorithms, depending on the data layout
- could be decoded later on via TFastReader.ReadVarUInt32Array
procedure WriteVarUInt64(Value: QWord);
Append
an unsigned integer value using 64-bit variable-length encoding
procedure WriteVarUInt64DynArray(const Values: TInt64DynArray; ValuesCount: integer; Offset: boolean);
Append
UInt64 values using 64-bit variable length integer encoding
- if Offset is TRUE, then it will store
the difference between two values using 64-bit variable-length integer encoding (in this case, a fixed-sized record storage is also handled separately)
- could be decoded later on via TFastReader.ReadVarUInt64Array
procedure WriteXor(New, Old: PAnsiChar; Len: PtrInt; crc: PCardinal = nil);
Append
"New[0..Len-1] xor Old[0..Len-1]" bytes
- as used e.g. by ZeroCompressXor
/TSynBloomFilterDiff.SaveTo
property BufferPosition: PtrInt read fPos;
The current position in the internal buffer
property Stream: TStream read fStream;
The associated writing stream
property Tag: PtrInt read fTag write fTag;
Simple property used to store
some integer content
property TotalWritten: Int64 read GetTotalWritten;
Get the byte count written since last Flush
TMultiPart = record
Used by MultiPartFormDataDecode
() to return one item of its data
TMemoryMapText = class(TSynPersistent)
Able to read a UTF-8 text
file using memory map
- much faster than TStringList.LoadFromFile()
- will ignore any trailing UTF-8 BOM in the file content, but will not expect
one either
constructor Create(aFileContent: PUtf8Char; aFileSize: integer); overload;
Read an UTF-8 encoded text
file content
- every line beginning is stored into LinePointers
[]
- this overloaded constructor accept an existing memory buffer (some uncompressed data e.g.)
constructor Create(const aFileName: TFileName); overload;
Read an UTF-8 encoded text
file
- every line beginning is stored into LinePointers
[]
destructor Destroy; override;
Release the memory map
and internal LinePointers
[]
function LineContains(const aUpperSearch: RawUtf8; aIndex: integer): boolean; virtual;
Returns TRUE if the supplied text
is contained in the corresponding line
function LineSize(aIndex: integer): integer;
Retrieve the number of UTF-8 chars of the given line
- warning: no range check is performed about supplied index
function LineSizeSmallerThan(aIndex, aMinimalCount: integer): boolean;
Check if there is at least a given number of UTF-8 chars in the given line
- this is faster than LineSize
(aIndex)<aMinimalCount for big lines
procedure AddInMemoryLine(const aNewLine: RawUtf8); virtual;
Add a new line to the already parsed content
- this line won't be stored in the memory mapped file, but stay in memory and appended to the existing lines
, until this instance is released
procedure AddInMemoryLinesClear; virtual;
Clear all in-memory appended rows
procedure SaveToFile(FileName: TFileName; const Header: RawUtf8 = '');
Save the whole content into a specified file
- including any runtime appended values via AddInMemoryLine
()
- an optional header text
can be added to the beginning of the file
procedure SaveToStream(Dest: TStream; const Header: RawUtf8);
Save the whole content into a specified stream
- including any runtime appended values via AddInMemoryLine
()
property Count: integer read fCount;
property FileName: TFileName read fFileName write fFileName;
The file name which was opened by this instance
property LinePointers: PPointerArray read fLines;
Direct access to each text
line
- use LineSize
() method to retrieve line length, since end of line will NOT end with #0, but with #13 or #10
- warning: no range check is performed about supplied index
property Lines[aIndex: integer]: RawUtf8 read GetLine;
Retrieve a line content as UTF-8
- a temporary UTF-8 string is created
- will return '' if aIndex is out of range
property Map: TMemoryMap read fMap;
The memory map
used to access the raw file content
property Strings[aIndex: integer]: string read GetString;
Retrieve a line content as RTL string type
- a temporary RTL string is created (after conversion for UNICODE Delphi)
- will return '' if aIndex is out of range
TProgressInfo = object(TObject)
Information about the progression of a process, e.g. for TStreamRedirect
- can also compute user-level text
information from raw numbers
Context: RawUtf8;
Optional process context
, e.g. a download URI, used for logging/progress
CurrentSize: Int64;
Number of bytes for the processed size
Elapsed: Int64;
Number of milliseconds elasped since process beginning
ExpectedSize: Int64;
Number of bytes for the final processed size
- may equal 0 if not known
OnLog: TSynLogProc;
Can be assigned from TSynLog.DoLog
class method for low-level logging
- at least at process startup and finish, and every second (ReportDelay
)
OnProgress: TOnInfoProgress;
Can be assigned to a TOnInfoProgress
callback for high-level logging
- at least at process startup and finish, and every second (ReportDelay
)
Percent: integer;
Percentage of CurrentSize
versus ExpectedSize
- equals 0 if ExpectedSize
is 0
PerSecond: PtrInt;
Number of bytes processed per second
ProcessedSize: Int64;
How many bytes have processed yet
Remaining: Int64;
Number of milliseconds remaining
for full process, as estimated
- equals 0 if ExpectedSize
is 0
- is just an estimation based on the average PerSecond
speed
ReportDelay: Int64;
Number of milliseconds between each DoReport
notification
- default is 1000, i.e. to notify OnLog
/OnProgress
every second
function DoReport(Sender: TObject; ReComputeElapsed: boolean): boolean;
Update the computed fields according to the curent state
- will be updated only every ReportDelay
ms (default 1000 = every second)
- return false and compute nothing if ReportDelay
has not been reached
- optionally call OnLog
and OnProgress
callbacks
function GetProgress: RawUtf8;
Retrieve the current status as simple text
- ready to be displayed on the console, e.g. as a single short line with no CRLF
during process, and eventually with full information and ending CRLF
procedure DoAfter(Sender: TObject; ChunkSize: Int64);
Can be called
procedure DoStart(Sender: TObject; SizeExpected: Int64; const Ident: string);
Initialize the information for a new process
- once expected size and ident are set, caller should call DoAfter
()
procedure Init;
Initialize the information, especially start the timing
procedure SetExpectedSize(SizeExpected, Position: Int64);
Called during process to setup ExpectedSize
/ExpectedWrittenSize fields
EStreamRedirect = class(ESynException)
Exception raised during TStreamRedirect
processing
TStreamRedirect = class(TStreamWithPosition)
An abstract pipeline stream able to redirect and hash
read/written content
- can be used either Read() or Write() calls during its livetime
- hashing is performed on the fly during the Read/Write process
- it features also a callback to mark its progress
- can sleep during Read/Write to reach a LimitPerSecond average bandwidth
constructor Create(aRedirected: TStream; aRead: boolean = false); reintroduce; virtual;
Initialize the internal structure, and start the timing
- before calling Read/Write, you should set the Redirected
property or specify aRedirected here - which will be owned by this instance
- if aRead is true, ExpectedSize
is set from aRedirected.Size
destructor Destroy; override;
Release the associated Redirected
stream
function GetHash: RawUtf8; virtual;
Return the current state of the hash
as lower hexadecimal
- by default, will return '' meaning that no hashing algorithm was set
class function GetHashFileExt: RawUtf8; virtual;
Current algorithm name as file/url extension, e.g. '.md5
' or '.sha256
'
- by default, will return '' meaning that no hashing algorithm was set
class function GetHashName: RawUtf8;
Current algorithm name, from GetHashFileExt
, e.g. 'md5
' or 'sha256
'
class function HashFile(const FileName: TFileName; const OnProgress: TOnStreamProgress = nil): RawUtf8;
Apply the internal hash
algorithm to the supplied file content
- could be used ahead of time to validate a cached file
function Read(var Buffer; Count: Longint): Longint; override;
Update the hash
and redirect the data to the associated TStream
- also trigger OnProgress
at least every second
- will raise an error if Write() (or Append
) have been called before
function Seek(const Offset: Int64; Origin: TSeekOrigin): Int64; override;
Overriden to support Seek
(0, soBeginning) and reset the Redirected
stream
- mandatory for proper THttpClientSocket.SockSendStream rewind
function Write(const Buffer; Count: Longint): Longint; override;
Update the hash
and redirect the data to the associated TStream
- also trigger OnProgress
at least every second
- will raise an error if Read() has been called before
procedure Append;
Update the hash
of the existing Redirected
stream content
- ready to Write() some new data after the existing
procedure Ended;
Notify end of process
- should be called explicitly when all Read()/Write() has been done
class procedure NotifyEnded( const OnStream: TOnStreamProgress; const OnInfo: TOnInfoProgress; const Fmt: RawUtf8; const Args: array of const; Size, StartedMs: Int64);
Notify a TOnStreamProgress
callback that a process ended
- create
a fake TStreamRedirect
and call Ended
with the supplied info
class procedure ProgressInfoToConsole(Sender: TObject; Info: PProgressInfo);
Can be used as TOnInfoProgress
callback writing into the console
class procedure ProgressStreamToConsole(Sender: TStreamRedirect);
Can be used as TOnStreamProgress
callback writing into the console
procedure Terminate;
Could be set from another thread to abort the streaming process
- will raise an exception at the next Read()/Write() call
property Context: RawUtf8 read fInfo.Context write fInfo.Context;
Optional process context
, e.g. a download URI, used for logging/progress
property Elapsed: Int64 read fInfo.Elapsed;
Number of milliseconds elasped since beginning, as set by Read/Write
property ExpectedSize: Int64 read fInfo.ExpectedSize write SetExpectedSize;
You can specify a number of bytes for the final Redirected
size
- will be used for the callback progress
- could be left to 0 for Write() if size is unknown
property LimitPerSecond: PtrInt read fLimitPerSecond write fLimitPerSecond;
Can limit the Read/Write bytes-per-second bandwidth used, if not 0
- sleep so that PerSecond
will keep close to this LimitPerSecond
value
property OnInfoProgress: TOnInfoProgress read fInfo.OnProgress write fInfo.OnProgress;
Optional TOnInfoProgress
callback triggered during Read/Write
- at least at process startup and finish, and every second / ReportDelay
property OnLog: TSynLogProc read fInfo.OnLog write fInfo.OnLog;
Can be assigned from TSynLog.DoLog
class method for low-level logging
property OnProgress: TOnStreamProgress read fOnStreamProgress write fOnStreamProgress;
Optional TOnStreamProgress
callback triggered during Read/Write
- at least at process startup and finish, and every second / ReportDelay
property Percent: integer read fInfo.Percent;
Percentage of Size versus ExpectedSize
- equals 0 if ExpectedSize
is 0
property PerSecond: PtrInt read fInfo.PerSecond;
Number of bytes processed per second, since initialization of this instance
property ProcessedSize: Int64 read fInfo.ProcessedSize;
How many bytes have passed through Read() or Write()
- may not equal Size or Position after an Append
- e.g. on resumed download from partial file
property Progress: RawUtf8 read GetProgress;
The current progression as text
, as returned by ProgressStreamToConsole
property Redirected: TStream read fRedirected write fRedirected;
Specify a TStream to which any Read()/Write() will be redirected
- this TStream instance will be owned by the TStreamRedirect
property Remaining: Int64 read fInfo.Remaining;
Number of milliseconds remaining
for full process, as set by Read/Write
- equals 0 if ExpectedSize
is 0
- is just an estimation based on the average PerSecond
speed
property ReportDelay: Int64 read fInfo.ReportDelay write fInfo.ReportDelay;
Number of milliseconds between each notification
- default is 1000, i.e. notify OnLog
/OnProgress
/OnInfoProgress
every second
property TimeOut: Int64 read fTimeOut write fTimeOut;
Read/Write will raise an exception if not finished after TimeOut
milliseconds
TStreamRedirectHash32 = class(TStreamRedirect)
TStreamRedirect
with 32-bit THasher
checksum
TStreamRedirectCrc32c = class(TStreamRedirectHash32)
TStreamRedirect
with crc32c
32-bit checksum
TFakeWriterStream = class(TStream)
A fake TStream, which will just count the number of bytes written
TNestedStreamReader = class(TStreamWithPositionAndSize)
TStream allowing to read from some nested TStream instances
destructor Destroy; override;
Finalize the nested TStream instance
function ForText: TRawByteStringStream;
Get the last TRawByteStringStream
, or append
a new one if needed
function NewStream(Stream: TStream): TStream;
Append
a nested TStream instance
- you could use a TFileStreamEx
here for efficient chunked reading
function Read(var Buffer; Count: Longint): Longint; override;
Will read up to Count bytes from the internal nested TStream
function Seek(const Offset: Int64; Origin: TSeekOrigin): Int64; override;
Overriden method to call Flush
on rewind, i.e. if position is set to 0
function Write(const Buffer; Count: Longint): Longint; override;
This TStream is read-only: calling this method will raise an exception
procedure Append(const Content: RawByteString);
Append
some text
or content to an internal TRawByteStringStream
- is the easy way to append
some text
or data to the internal buffers
procedure Flush; virtual;
You should call this method before any Read() call
- is also called when you execute Seek
(0, soBeginning)
TBufferedStreamReader = class(TStreamWithPositionAndSize)
TStream with an internal memory buffer
- can be beneficial e.g. reading from a file by small chunks
constructor Create(const aSourceFileName: TFileName; aBufSize: integer = 65536); reintroduce; overload;
Initialize a source file and the internal buffer
constructor Create(aSource: TStream; aBufSize: integer = 65536); reintroduce; overload;
Initialize the source TStream and the internal buffer
- will also rewind the aSource position to its beginning, and retrieve its size
destructor Destroy; override;
Finalize this instance and its buffer
function Read(var Buffer; Count: Longint): Longint; override;
Will read up to Count bytes from the internal buffer or source TStream
function Seek(const Offset: Int64; Origin: TSeekOrigin): Int64; override;
Overriden method to flush buffer on rewind
function Write(const Buffer; Count: Longint): Longint; override;
This TStream is read-only: calling this method will raise an exception
TRawByteStringGroupValue = record
Item as stored in a TRawByteStringGroup
instance
TRawByteStringGroup = object(TObject)
Store
several RawByteString
content with optional concatenation
Count: integer;
How many items are currently stored in Values
[]
LastFind: integer;
Naive but efficient cache for Find
()
Position: integer;
The current size of data stored in Values
[]
Values: TRawByteStringGroupValueDynArray;
Actual list storing the data
function AsBytes: TByteDynArray;
Return all content as a single TByteDynArray
function AsText: RawByteString;
Return all content as a single RawByteString
- will also compact
the Values
[] array into a single item (which is returned)
function Equals(const aAnother: TRawByteStringGroup): boolean;
Compare two TRawByteStringGroup
instance stored text
function Find(aPosition: integer): PRawByteStringGroupValue; overload;
Returns a pointer to Values
[] containing a given position
- returns nil if not found
function Find(aPosition, aLength: integer): pointer; overload;
Returns a pointer to Values
[].Value containing a given position
and length
- returns nil if not found
function FindAsText(aPosition, aLength: integer): RawByteString; overload;
Returns the text
at a given position
in Values
[]
- text
should be in a single Values
[] entry
procedure Add(const aItem: RawByteString); overload;
procedure Add(const aAnother: TRawByteStringGroup); overload;
Add
another TRawByteStringGroup
to Values
[]
procedure Add(aItem: pointer; aItemLen: integer); overload;
procedure AddFromReader(var aReader: TFastReader);
Add
another TRawByteStringGroup
previously serialized via WriteString
()
procedure AppendTextAndClear(var aDest: RawByteString);
Append
stored information into another RawByteString
, and clear
content
procedure Clear;
Clear
any stored information
procedure Compact;
Compact
the Values
[] array into a single item
- is also used by AsText
to compute a single RawByteString
procedure FindAsText(aPosition, aLength: integer; out aText: RawByteString); overload;
Returns the text
at a given position
in Values
[]
- text
should be in a single Values
[] entry
procedure FindAsVariant(aPosition, aLength: integer; out aDest: variant);
Returns the text
at a given position
in Values
[] via RawUtf8ToVariant
()
- text
should be in a single Values
[] entry
- explicitly returns null
if the supplied text
was not found
procedure FindMove(aPosition, aLength: integer; aDest: pointer);
Copy the text
at a given position
in Values
[]
- text
should be in a single Values
[] entry
procedure FindWrite(aPosition, aLength: integer; W: TTextWriter; Escape: TTextWriterKind = twJsonEscape; TrailingCharsToIgnore: integer = 0);
Append
the text
at a given position
in Values
[], JSON escaped by default
- text
should be in a single Values
[] entry
procedure FindWriteBase64(aPosition, aLength: integer; W: TTextWriter; withMagic: boolean);
Append
the blob at a given position
in Values
[], Base64 encoded
- text
should be in a single Values
[] entry
procedure RemoveLastAdd;
Low-level method to abort the latest Add
() call
- warning: will work only once, if an Add
() has actually been just called: otherwise, the behavior is unexpected, and may wrongly truncate data
procedure Write(W: TTextWriter; Escape: TTextWriterKind = twJsonEscape); overload;
Save all content into a TJsonWriter
instance
procedure WriteBinary(W: TBufferWriter); overload;
Save all content into a TBufferWriter
instance
procedure WriteString(W: TBufferWriter);
Save all content as a string into a TBufferWriter
instance
- storing the length as WriteVarUInt32() prefix
TRawByteStringHeap = class(TObject)
Thread-safe reusable set of constant RawByteString
/RawUtf8
instances
- all RawByteString
will be constant (RefCnt=-2) with some max length
- maintain internally its own TLockedList
O(1) set of instances
- warning: any call to New() should manually be followed by one Release()
constructor Create(aMaxLength: integer);
Initialize the internal cache for a given maximum length
destructor Destroy; override;
Release
all cached instances
function Clean: PtrInt;
Release
the RawByteString
instances in the cache bin
- keep any existing New
() instances intact
- returns how many memory has been released to the heap
procedure New(var aDest: RawByteString; aText: PUtf8Char = nil; aTextLen: TStrLen = 0; aCodePage: integer = CP_RAWBYTESTRING); overload;
Return a new
RawByteString
of a given length, with refcount=-2
- if aTextLen is its default 0, MaxLength
will be used
- returned from its internal cache, unless aTextLen>MaxLength
, which will allocate a regular RawByteString
from heap
procedure New(var aDest: RawUtf8; aText: PUtf8Char; aTextLen: TStrLen); overload;
Return a new
RawUtf8
of a given length, with refcount = -2
procedure Release(var aDest: pointer); overload;
Put back a RawByteString
acquired from NewUtf8() into the internal cache
procedure Release(var aDest: RawByteString); overload;
Put back a RawByteString
acquired from New
() into the internal cache
procedure Release(var aDest: RawUtf8); overload;
Put back a RawUtf8
acquired from New
() into the internal cache
property Count: integer read fOne.Count;
How many New
() calls are currently active
property MaxLength: TStrLen read fMaxLength;
The maximum length() of RawByteString
returned by New
()
TRawByteStringBuffer = object(TObject)
Store
one RawByteString
content with an associated length
- to be used e.g. as a convenient reusable memory buffer
function Buffer: pointer;
A convenient wrapper to pointer(fBuffer) for direct Buffer
/Len
use
function Capacity: PtrInt;
How many bytes are currently allocated in the Buffer
function Extract(Dest: pointer; Count: PtrInt): PtrInt;
Move up to Count bytes from the internal Buffer
into another place
- returns how many bytes were available to be copied into Dest^
- then remove
the copied bytes from the internal Buffer
/Len
storage
function ExtractAt(var Dest: PAnsiChar; var Count: PtrInt; var Pos: PtrInt): PtrInt;
Move up to Count bytes from the internal Buffer
into another place
- returns how many bytes were available to be copied into Dest^
- don't move any byte, but just update the given Pos index
function TryAppend(P: pointer; PLen: PtrInt): boolean;
Add some UTF-8 buffer
content to the Buffer
, without resizing it
procedure Append(const Text: array of RawUtf8); overload;
Add some UTF-8 string(s) content to the Buffer
, resizing it if needed
procedure Append(Ch: AnsiChar); overload;
Just after Append
/AppendShort
, append
one single character
procedure Append(const Text: RawUtf8); overload;
Add some UTF-8 string content to the Buffer
, resizing it if needed
procedure Append(P: pointer; PLen: PtrInt); overload;
Add some UTF-8 buffer
content to the Buffer
, resizing it if needed
procedure Append(Value: QWord); overload;
Add some number as text
content to the Buffer
, resizing it if needed
procedure AppendCRLF;
Just after Append
/AppendShort
, append
a #13#10 end of line
procedure AppendShort(const Text: ShortString);
Add some UTF-8 shortstring content to the Buffer
, resizing it if needed
procedure AsText(out Text: RawUtf8; Overhead: PtrInt = 0; UseMainBuffer: boolean = false);
Retrieve the current Buffer
/Len
content as RawUtf8
text
- with some optional overhead for faster reallocmem at concatenation
- won't force Len
to 0: caller should call Reset
if done with it
- UseMainBuffer=true will return a copy of fBuffer into Text
procedure Clear;
Release/free the internal Buffer
storage
procedure Remove(FirstBytes: PtrInt);
Similar to delete(fBuffer, 1, FirstBytes)
procedure Reserve(MaxSize: PtrInt); overload;
Ensure the internal Buffer
has at least MaxSize bytes
- also reset
the internal Len
to 0
procedure Reserve(const WorkingBuffer: RawByteString); overload;
Use a specified string buffer
as start
procedure Reset;
Set Len
to 0, but doesn't clear
/free the Buffer
itself
property Len: PtrInt read fLen write fLen;
How many bytes are currently used in the Buffer
PRawByteStringBuffer = ^TRawByteStringBuffer;
Pointer reference to a TRawByteStringBuffer
PRawByteStringGroup = ^TRawByteStringGroup;
Pointer reference to a TRawByteStringGroup
TAlgoCompressLoad = ( aclNormal, aclSafeSlow, aclNoCrcFast );
Define the implementation used by TAlgoCompress.Decompress
()
TBufferWriterKind = ( wkUInt32, wkVarUInt32, wkVarInt32, wkSorted, wkOffsetU, wkOffsetI, wkFakeMarker );
Available kind of integer array storage, corresponding to the data layout of TBufferWriter
- wkUInt32 will write the content as "plain" 4 bytes binary (this is the prefered way if the integers can be negative)
- wkVarUInt32 will write the content using our 32-bit variable-length integer encoding
- wkVarInt32 will write the content using our 32-bit variable-length integer encoding and the by-two complement (0=0,1=1,2=-1,3=2,4=-2...)
- wkSorted will write an increasing array of integers, handling the special case of a difference of similar value (e.g. 1) between two values - note that this encoding is efficient only if the difference is mainly < 253
- wkOffsetU and wkOffsetI will write the difference between two successive values, with detection of any constant difference (unsigned or signed)
- wkFakeMarker won't be used by WriteVarUInt32Array, but to notify a custom encoding
TEmoji = ( eNone, eGrinning, eGrin, eJoy, eSmiley, eSmile, eSweat_smile, eLaughing, eInnocent, eSmiling_imp, eWink, eBlush, eYum, eRelieved, eHeart_eyes, eSunglasses, eSmirk, eNeutral_face, eExpressionless, eUnamused, eSweat, ePensive, eConfused, eConfounded, eKissing, eKissing_heart, eKissing_smiling_eyes, eKissing_closed_eyes, eStuck_out_tongue, eStuck_out_tongue_winking_eye, eStuck_out_tongue_closed_eyes, eDisappointed, eWorried, eAngry, ePout, eCry, ePersevere, eTriumph, eDisappointed_relieved, eFrowning, eAnguished, eFearful, eWeary, eSleepy, eTired_face, eGrimacing, eSob, eOpen_mouth, eHushed, eCold_sweat, eScream, eAstonished, eFlushed, eSleeping, eDizzy_face, eNo_mouth, eMask, eSmile_cat, eJoy_cat, eSmiley_cat, eHeart_eyes_cat, eSmirk_cat, eKissing_cat, ePouting_cat, eCrying_cat_face, eScream_cat, eSlightly_frowning_face, eSlightly_smiling_face, eUpside_down_face, eRoll_eyes, eNo_good, oOk_woman, eBow, eSee_no_evil, eHear_no_evil, eSpeak_no_evil, eRaising_hand, eRaised_hands, eFrowning_woman, ePerson_with_pouting_face, ePray );
Map the first Unicode page of Emojis, from U+1F600 to U+1F64F
- naming comes from github/Markdown :identifiers:
TFileBufferWriter = TBufferWriter;
Deprecated alias to TBufferWriter
binary serializer
THashFile = function(const FileName: TFileName): RawUtf8;
Prototype of a file hashing function, returning its hexadecimal hash
- match HashFileCrc32c
() below, HashFileCrc32
() in mormot.core.zip, and HashFileMd5
/HashFileSha* in mormot.crypt.secure functions signature
TLogEscape = array[0..511] of AnsiChar;
512 bytes buffer to be allocated on stack when using LogEscape
()
TMimeType = ( mtUnknown, mtPng, mtGif, mtTiff, mtJpg, mtBmp, mtDoc, mtPpt, mtXls, mtHtml, mtCss, mtJS, mtXIcon, mtFont, mtText, mtSvg, mtXml, mtWebp, mtManifest, mtJson, mtOgg, mtMp4, mtMp2, mtMpeg, mtH264, mtWma, mtWmv, mtAvi, mtGzip, mtWebm, mtRar, mt7z, mtZip, mtBz2, mtPdf, mtSQlite3, mtXcomp, mtDicom );
Some of the best-known mime types
- subset of the whole IANA list which can be quite huge (>1500 items)
TMultiPartDynArray = array of TMultiPart;
Used by MultiPartFormDataDecode
() to return all its data items
TOnFastReaderErrorData = procedure(const fmt: RawUtf8; const args: array of const) of object;
Event signature to customize TFastReader.ErrorData
notification
TOnFastReaderErrorOverflow = procedure of object;
Event signature to customize TFastReader.ErrorOverflow
notification
TOnInfoProgress = procedure(Sender: TObject; Info: PProgressInfo) of object;
Callback definition to notify some TProgressInfo
process
- see e.g. TStreamRedirect.ProgressInfoToConsole
TOnStreamCreate = function(const FileName: string; Mode: cardinal): TStream of object;
Optional callback as used e.g. by THttpClientSocketWGet.OnStreamCreate
TOnStreamProgress = procedure(Sender: TStreamRedirect) of object;
TStreamHasher.Write optional progression callback
- see Sender properties like Context/Size/PerSecond and ExpectedSize (which may be 0 if the download size is unknown)
- see e.g. TStreamRedirect.ProgressStreamToConsole
TRawByteStringGroupValueDynArray = array of TRawByteStringGroupValue;
Items as stored in a TRawByteStringGroup
instance
TStreamRedirectClass = class of TStreamRedirect;
Meta-class of TStreamRedirect
hierarchy
TTextWriterHtmlEscape = set of ( heHtmlEscape, heEmojiToUtf8);
Tune AddHtmlEscapeWiki
/AddHtmlEscapeMarkdown
wrapper functions process
- heHtmlEscape will escape any HTML special chars, e.g. & into &
- heEmojiToUtf8 will convert any Emoji text
into UTF-8 Unicode character, recognizing e.g. :joy: or :) in the text
TUrlEncoder = set of ( ueTrimLeadingQuestionMark, ueEncodeNames, ueStarNameIsCsv, ueSkipVoidString, ueSkipVoidValue);
Some options for UrlEncode
()
ALGO_SAFE: array[boolean] of TAlgoCompressLoad = ( aclNormal, aclSafeSlow);
Used e.g. as when ALGO_SAFE
[SafeDecompression] for TAlgoCompress.Decompress
JSON_BASE64_MAGIC_C = $b0bfef;
UTF-8 encoded \uFFF0 special code to mark Base64 binary content in JSON
- Unicode special char U+FFF0 is UTF-8 encoded as EF BF B0 bytes
- as generated by BinToBase64WithMagic
() functions, and expected by the TExtractInlineParameters
decoder
- used e.g. when transmitting TDynArray.SaveTo
() content
JSON_BASE64_MAGIC_QUOTE_C = ord('"') + cardinal(JSON_BASE64_MAGIC_C) shl 8;
'"' + UTF-8 encoded \uFFF0 special code to mark Base64 binary in JSON
JSON_BASE64_MAGIC_QUOTE_S: string[4] = '"'#$ef#$bf#$b0;
'"' + UTF-8 encoded \uFFF0 special code to mark Base64 binary in JSON
- defined as a ShortString constant to be used as:
AddShorter(JSON_BASE64_MAGIC_QUOTE_S);
JSON_BASE64_MAGIC_S: string[3] = #$ef#$bf#$b0;
UTF-8 encoded \uFFF0 special code to mark Base64 binary content in JSON
- Unicode special char U+FFF0 is UTF-8 encoded as EF BF B0 bytes
- as generated by BinToBase64WithMagic
() functions, and expected by the TExtractInlineParameters
decoder
- used e.g. when transmitting TDynArray.SaveTo
() content
LOG_MAGIC = $ABA51051;
The "magic" number used to identify .log.synlz compressed files, as created by EventArchiveSynLZ
/ EventArchiveLizard
callbacks
MIME_TYPE: array[TMimeType] of RawUtf8 = ( '', 'image/png', 'image/gif', 'image/tiff', JPEG_CONTENT_TYPE, 'image/bmp', 'application/msword', 'application/vnd.ms-powerpoint', 'application/vnd.ms-excel', HTML_CONTENT_TYPE, 'text/css', 'text/javascript', 'image/x-icon', 'font/woff', TEXT_CONTENT_TYPE, 'image/svg+xml', XML_CONTENT_TYPE, 'image/webp', 'text/cache-manifest', JSON_CONTENT_TYPE, 'video/ogg', 'video/mp4', 'video/mp2', 'audio/mpeg', 'video/H264', 'audio/x-ms-wma', 'video/x-ms-wmv', 'video/x-msvideo', 'application/gzip', 'video/webm', 'application/x-rar-compressed', 'application/x-7z-compressed', 'application/zip', 'application/bzip2', 'application/pdf', 'application/x-sqlite3', 'application/x-compress', 'application/dicom');
The known mime types text
representation
PLURAL_FORM: array[boolean] of RawUtf8 = ( '', 's');
Can be used to append
to most English nouns to form a plural
- as used by the Plural
() function
SYNLZTRIG: array[boolean] of integer = ( 100, maxInt);
CompressionSizeTrigger parameter SYNLZTRIG
[true] will disable then SynLZCompress
() compression
Functions or procedures | Description | |
---|---|---|
AddHtmlEscapeMarkdown | Convert minimal Markdown text into proper HTML | |
AddHtmlEscapeWiki | Convert some wiki-like text into proper HTML | |
Append999ToBuffer | Fast add text conversion of 0-999 integer value into a given buffer | |
AppendBuffersToRawUtf8 | Fast add some characters to a RawUtf8 string | |
AppendBufferToBuffer | Fast add some characters from ane buffer into another buffer | |
AppendBufferToRawByteString | Just redirect to mormot.core.text Append(...) overloads | |
AppendCharOnceToRawUtf8 | Fast add one character to a RawUtf8 string, if not already present | |
AppendRawUtf8ToBuffer | Fast add some characters from a RawUtf8 string into a given buffer | |
AppendUInt32ToBuffer | Fast add text conversion of a 32-bit signed integer value into a given buffer | |
AreUrlValid | Checks if the supplied UTF-8 text values don't need URI encoding | |
AsciiToBaudot | Convert some ASCII-7 text into binary, using Emile Baudot code | |
AsciiToBaudot | Convert some ASCII-7 text into binary, using Emile Baudot code | |
Base32ToBin | Conversion from Base32 encoded text into a binary string | |
Base32ToBin | Conversion from Base32 encoded text into a binary string | |
Base58ToBin | Conversion from Base58 encoded text into a binary string | |
Base58ToBin | Conversion from Base58 encoded text into a binary string | |
Base58ToBin | Conversion from Base58 encoded text into a binary buffer | |
Base64Decode | Direct low-level decoding of a Base64 encoded buffer | |
Base64Encode | Raw function for efficient binary to Base64 encoding | |
Base64EncodeTrailing | Raw function for efficient binary to Base64 encoding of the last bytes | |
Base64MagicCheckAndDecode | Check and decode '\uFFF0base64encodedbinary' content into binary | |
Base64MagicCheckAndDecode | Check and decode '\uFFF0base64encodedbinary' content into binary | |
Base64MagicCheckAndDecode | Check and decode '\uFFF0base64encodedbinary' content into binary | |
Base64MagicDecode | Just a wrapper around Base64ToBin() for in-place decode of JSON_BASE64_MAGIC_C '\uFFF0base64encodedbinary' content into binary | |
Base64MagicToBlob | Convert a Base64-encoded content into binary hexadecimal ready for SQL | |
Base64MagicTryAndDecode | Decode '\uFFF0base64encodedbinary' or 'base64encodedbinary' into binary | |
Base64ToBin | Fast conversion from Base64 encoded text into binary data | |
Base64ToBin | Fast conversion from Base64 encoded text into binary data | |
Base64ToBin | Fast conversion from Base64 encoded text into binary data | |
Base64ToBin | Fast conversion from Base64 encoded text into binary data | |
Base64ToBin | Fast conversion from Base64 encoded text into binary data | |
Base64ToBin | Fast conversion from Base64 encoded text into binary data | |
Base64ToBinLength | Retrieve the expected undecoded length of a Base64 encoded buffer | |
Base64ToBinLengthSafe | Retrieve the expected undecoded length of a Base64 encoded buffer | |
Base64ToBinSafe | Fast conversion from Base64 encoded text into binary data | |
Base64ToBinSafe | Fast conversion from Base64 encoded text into binary data | |
Base64ToBinSafe | Fast conversion from Base64 encoded text into binary data | |
Base64ToBinSafe | Fast conversion from Base64 encoded text into binary data | |
Base64ToBinTrim | Conversion from Base64 encoded text into binary data, ignoring spaces | |
Base64ToUri | Conversion from any Base64 encoded value into URI-compatible encoded text | |
Base64uriDecode | Direct low-level decoding of a Base64-URI encoded buffer | |
Base64uriEncode | Low-level conversion from a binary buffer into Base64-like URI-compatible encoded text | |
Base64uriToBin | Fast conversion from Base64-URI encoded text into binary data | |
Base64uriToBin | Fast conversion from Base64-URI encoded text into binary data | |
Base64uriToBin | Fast conversion from Base64-URI encoded text into binary data | |
Base64uriToBin | Fast conversion from Base64-URI encoded text into binary data | |
Base64uriToBin | Fast conversion from Base64-URI encoded text into binary data | |
Base64uriToBin | Fast conversion from Base64-URI encoded text into binary data | |
Base64uriToBinLength | Retrieve the expected undecoded length of a Base64-URI encoded buffer | |
BaudotToAscii | Convert some Baudot code binary, into ASCII-7 text | |
BaudotToAscii | Convert some Baudot code binary, into ASCII-7 text | |
BinToBase32 | Conversion from a binary buffer into Base32 encoded text as RawUtf8 | |
BinToBase32 | Conversion from a binary buffer into Base32 encoded text as RawUtf8 | |
BinToBase32 | Conversion from a binary buffer into Base32 encoded text buffer | |
BinToBase32Length | Compute the length resulting of Base32 encoding of a binary buffer | |
BinToBase58 | Conversion from a binary buffer into Base58 encoded text as TSynTempBuffer | |
BinToBase58 | Conversion from a binary buffer into Base58 encoded text as RawUtf8 | |
BinToBase58 | Conversion from a binary buffer into Base58 encoded text as RawUtf8 | |
BinToBase64 | Fast conversion from binary data into prefixed/suffixed Base64 encoded UTF-8 text | |
BinToBase64 | Fast conversion from binary data into Base64 encoded UTF-8 text | |
BinToBase64 | Fast conversion from binary data into Base64 encoded UTF-8 text | |
BinToBase64Length | Retrieve the expected encoded length after Base64 process | |
BinToBase64Line | Fast conversion from binary into prefixed/suffixed Base64 with 64 chars per line | |
BinToBase64Short | Fast conversion from a small binary data into Base64 encoded UTF-8 text | |
BinToBase64Short | Fast conversion from a small binary data into Base64 encoded UTF-8 text | |
BinToBase64uri | Fast conversion from binary data into Base64-like URI-compatible encoded text | |
BinToBase64uri | Fast conversion from a binary buffer into Base64-like URI-compatible encoded text | |
BinToBase64uriLength | Retrieve the expected encoded length after Base64-URI process | |
BinToBase64uriShort | Fast conversion from a binary buffer into Base64-like URI-compatible encoded ShortString | |
BinToBase64WithMagic | Fast conversion from binary data into Base64 encoded UTF-8 text with JSON_BASE64_MAGIC_C prefix (UTF-8 encoded \uFFF0 special code) | |
BinToBase64WithMagic | Fast conversion from binary data into Base64 encoded UTF-8 text with JSON_BASE64_MAGIC_C prefix (UTF-8 encoded \uFFF0 special code) | |
BinToBase64WithMagic | Fast conversion from binary data into Base64 encoded UTF-8 text with JSON_BASE64_MAGIC_C prefix (UTF-8 encoded \uFFF0 special code) | |
BinToHumanHex | Generate some 'xx:xx:xx:xx' output buffer with left and right margins | |
BinToHumanHex | Generate some 'xx:xx:xx:xx' output buffer with left and right margins | |
BinToSource | Generate some pascal source code holding some data binary as constant | |
BinToSource | Generate some pascal source code holding some data binary as constant | |
BinToSource | Generate some pascal source code holding some data binary as constant | |
BlobToBytes | Create a TBytes from TEXT-encoded blob data | |
BlobToRawBlob | Fill a RawBlob from TEXT-encoded blob data | |
BlobToRawBlob | Fill a RawBlob from TEXT-encoded blob data | |
BlobToRawBlob | Fill a RawBlob from TEXT-encoded blob data | |
BlobToStream | Create a memory stream from TEXT-encoded blob data | |
BytesToRawByteString | Creates a RawByteString memory buffer from a TBytes content | |
ContentToShort | If source is not UTF-8 calls EscapeToShort, otherwise return it directly | |
EmojiFromDots | Low-level conversion of github/Markdown :identifiers: into UTF-8 Emoji sequences | |
EmojiFromDots | Conversion of github/Markdown :identifiers: into UTF-8 Emoji sequences | |
EmojiFromText | Recognize github/Markdown compatible text of Emojis | |
EmojiParseDots | Low-level parser of github/Markdown compatible text of Emojis | |
EmojiToDots | Low-level conversion of UTF-8 Emoji sequences into github/Markdown :identifiers: | |
EmojiToDots | Conversion of UTF-8 Emoji sequences into github/Markdown :identifiers: | |
EscapeBuffer | Low-level fast conversion from binary data to escaped text | |
EscapeToShort | Fill a ShortString with the (hexadecimal) chars of the input text/binary | |
EscapeToShort | Fill a ShortString with the (hexadecimal) chars of the input text/binary | |
FromVarBlob | Retrieve pointer and length to a variable-length text/blob buffer | |
FromVarInt32 | Convert a 32-bit variable-length integer buffer into an integer | |
FromVarInt64 | Convert a 64-bit variable-length integer buffer into a Int64 | |
FromVarInt64Value | Convert a 64-bit variable-length integer buffer into a Int64 | |
FromVarString | Retrieve a variable-length UTF-8 encoded text buffer in a newly allocation RawUtf8 | |
FromVarString | Retrieve a variable-length UTF-8 encoded text buffer in a temporary buffer | |
FromVarString | Retrieve a variable-length UTF-8 encoded text buffer in a temporary buffer | |
FromVarString | Safe retrieve a variable-length UTF-8 encoded text buffer in a newly allocation RawUtf8 | |
FromVarString | Retrieve a variable-length text buffer | |
FromVarString | Retrieve a variable-length text buffer | |
FromVarString | Retrieve a variable-length UTF-8 encoded text buffer in a newly allocation RawUtf8 | |
FromVarUInt32 | Convert a 32-bit variable-length integer buffer into a cardinal | |
FromVarUInt32 | Convert a 32-bit variable-length integer buffer into a cardinal | |
FromVarUInt32Big | Convert a 32-bit variable-length integer buffer into a cardinal | |
FromVarUInt32High | Convert a 32-bit variable-length integer buffer into a cardinal | |
FromVarUInt32Safe | Safely convert a 32-bit variable-length integer buffer into a cardinal | |
FromVarUInt32Up128 | Convert a 32-bit variable-length integer buffer into a cardinal | |
FromVarUInt64 | Convert a 64-bit variable-length integer buffer into a UInt64 | |
FromVarUInt64 | Convert a 64-bit variable-length integer buffer into a UInt64 | |
FromVarUInt64Safe | Safely convert a 64-bit variable-length integer buffer into a UInt64 | |
GetJpegSize | Fast guess of the size, in pixels, of a JPEG memory buffer | |
GetMimeContentType | Retrieve the MIME content type from its file name or a supplied binary buffer | |
GetMimeContentTypeFromBuffer | Retrieve the MIME content type from a supplied binary buffer | |
GetMimeContentTypeFromExt | MtUnknown mtPng mtGif mtTiff mtJpg mtBmp mtDoc mtPpt mtXls mtHtml mtCss mtJS RFC 9239 mtXIcon mtFont RFC 8081 mtText mtSvg mtXml mtWebp mtManifest mtJson mtOgg RFC 5334 mtMp4 RFC 4337 6381 mtMp2 mtMpeg RFC 3003 mtH264 RFC 6184 mtWma mtWmv mtAvi mtGzip mtWebm mtRar mt7z mtZip mtBz2 mtPdf mtSQlite3 mtXcomp mtDicom retrieve the MIME content type from its file name | |
GetMimeContentTypeFromMemory | Retrieve the MIME content type from a supplied binary buffer | |
GetMimeContentTypeHeader | Retrieve the HTTP header for MIME content type from a supplied binary buffer | |
GetMimeTypeFromExt | Retrieve the MIME content type from its file extension text (without '.') | |
GetStreamBuffer | Retrieve the memory buffer of a TCustomMemoryStream/TRawByteStringStream | |
GotoNextVarInt | Jump a value in the 32-bit or 64-bit variable-length integer buffer | |
GotoNextVarString | Jump a value in variable-length text buffer | |
HashFile | Compute the 32-bit default hash of a file content | |
HashFileCrc32c | Compute the crc32c checksum of a given file | |
HtmlEscapeMarkdown | Escape some Markdown-marked text into HTML | |
HtmlEscapeWiki | Escape some wiki-marked text into HTML | |
IncludeTrailingUriDelimiter | Ensure the supplied URI contains a trailing '/' charater | |
IsBase64 | Check if the supplied text is a valid Base64 encoded stream | |
IsBase64 | Check if the supplied text is a valid Base64 encoded stream | |
isBlobHex | Return true if the TEXT is encoded as SQLite3 BLOB literals (X'53514C697465' e.g.) | |
IsContentCompressed | Retrieve if some content is compressed, from a supplied binary buffer | |
IsContentTypeCompressible | Recognize e.g. 'text/css' or 'application/json' as compressible | |
IsContentTypeCompressibleU | Recognize e.g. 'text/css' or 'application/json' as compressible | |
IsContentTypeJson | Recognize e.g. 'application/json' or 'application/vnd.api+json' | |
IsStreamBuffer | Check if class is a TCustomMemoryStream/TRawByteStringStream | |
IsUrlValid | Checks if the supplied UTF-8 text don't need URI encoding | |
LogEscape | Fill TLogEscape stack buffer with the (hexadecimal) chars of the input binary | |
LogEscapeFull | Returns a text buffer with the (hexadecimal) chars of the input binary | |
LogEscapeFull | Returns a text buffer with the (hexadecimal) chars of the input binary | |
MultiPartFormDataAddField | Encode a field in a multipart array | |
MultiPartFormDataAddFile | Encode a file in a multipart array | |
MultiPartFormDataDecode | Decode multipart/form-data POST request content into memory | |
MultiPartFormDataEncode | Encode multipart fields and files | |
MultiPartFormDataNewBound | Used e.g. by MultiPartFormDataEncode and THttpMultiPartStream.Add | |
Plural | Write count number and append 's' (if needed) to form a plural English noun | |
RawBlobToBlob | Creates a TEXT-encoded version of blob data from a memory data | |
RawBlobToBlob | Creates a TEXT-encoded version of blob data from a RawBlob | |
RawByteStringArrayConcat | Fast concatenation of several AnsiStrings | |
RawByteStringToBytes | Creates a TBytes from a RawByteString memory buffer | |
ResourceSynLZToRawByteString | Creates a RawByteString memory buffer from an SynLZ-compressed embedded resource | |
ResourceToRawByteString | Creates a RawByteString memory buffer from an embedded resource | |
SameFileContent | Compare two files by content, reading them by blocks | |
ToVarInt32 | Convert an integer into a 32-bit variable-length integer buffer | |
ToVarInt64 | Convert a Int64 into a 64-bit variable-length integer buffer | |
ToVarString | Convert a RawUtf8 into an UTF-8 encoded variable-length buffer | |
ToVarUInt32 | Convert a cardinal into a 32-bit variable-length integer buffer | |
ToVarUInt32Length | Return the number of bytes necessary to store a 32-bit variable-length integer | |
ToVarUInt32LengthWithData | Return the number of bytes necessary to store some data with a its 32-bit variable-length integer length | |
ToVarUInt64 | Convert a UInt64 into a 64-bit variable-length integer buffer | |
UrlAppend | Append some encoded Name,Value pairs parameters to an existing URI | |
UrlDecode | Decode a UrlEncode() URI encoded parameter into its original value | |
UrlDecode | Decode a UrlEncode() URI encoded parameter into its original value | |
UrlDecodeCardinal | Decode a specified parameter compatible with URI encoding into its original cardinal numerical value | |
UrlDecodeDouble | Decode a specified parameter compatible with URI encoding into its original floating-point value | |
UrlDecodeExtended | Decode a specified parameter compatible with URI encoding into its original floating-point value | |
UrlDecodeInt64 | Decode a specified parameter compatible with URI encoding into its original Int64 numerical value | |
UrlDecodeInteger | Decode a specified parameter compatible with URI encoding into its original integer numerical value | |
UrlDecodeName | Decode a UrlEncodeName() URI encoded network name into its original value | |
UrlDecodeName | Decode a UrlEncodeName() URI encoded network name into its original value | |
UrlDecodeNeedParameters | Returns TRUE if all supplied parameters do exist in the URI encoded text | |
UrlDecodeNextName | Decode a URI-encoded Name from an input buffer | |
UrlDecodeNextNameValue | Decode the next Name=Value&.... pair from input URI | |
UrlDecodeNextValue | Decode a URI-encoded Value from an input buffer | |
UrlDecodeValue | Decode a specified parameter compatible with URI encoding into its original textual value | |
UrlDecodeVar | Decode a UrlEncode/UrlEncodeName() URI encoded string into its original value | |
UrlEncode | Encode a string as URI parameter encoding, i.e. ' ' as '+' | |
UrlEncode | Encode supplied parameters to be compatible with URI encoding | |
UrlEncode | Encode supplied parameters to be compatible with URI encoding | |
UrlEncode | Encode a string as URI parameter encoding, i.e. ' ' as '+' | |
UrlEncode | Append a string as URI parameter encoding, i.e. ' ' as '+' | |
UrlEncode | Append a string as URI parameter encoding, i.e. ' ' as '+' | |
UrlEncodeFull | Encode a full URI with prefix and parameters | |
UrlEncodeName | Encode a string as URI network name encoding, i.e. ' ' as %20 | |
UrlEncodeName | Encode a string as URI network name encoding, i.e. ' ' as %20 | |
UrlEncodeName | Append a string as URI network name encoding, i.e. ' ' as %20 | |
UrlEncodeName | Append a string as URI network name encoding, i.e. ' ' as %20 |
procedure AddHtmlEscapeMarkdown(W: TTextWriter; P: PUtf8Char; esc: TTextWriterHtmlEscape = [heEmojiToUtf8]);
Convert minimal Markdown text
into proper HTML
- see https://enterprise.github.com/downloads/en/markdown-cheatsheet.pdf
- convert all #13#10 into <p>...</p>, *..* into <em>..</em>, **..** into <strong>..</strong>, `...` into <code>...</code>, backslash espaces \\ \* \_ and so on, [title](http
://...) and detect plain http
:// as <a href=...>
- create unordered lists from trailing * + - chars, blockquotes from trailing > char, and code line from 4 initial spaces
- as with default Markdown, won't escape HTML special chars (i.e. you can write plain HTML in the supplied text
) unless esc is set otherwise
- only inline-style links and images are supported yet (not reference-style); tables aren't supported either
procedure AddHtmlEscapeWiki(W: TTextWriter; P: PUtf8Char; esc: TTextWriterHtmlEscape = [heHtmlEscape, heEmojiToUtf8]);
Convert some wiki-like text
into proper HTML
- convert all #13#10 into <p>...</p>, *..* into <em>..</em>, +..+ into <strong>..</strong>, `..` into <code>..</code>, and http
://... as <a href=http
://...>
- escape any HTML special chars, and Emoji tags as specified with esc
function Append999ToBuffer(Buffer: PUtf8Char; Value: PtrUInt): PUtf8Char;
Fast add text
conversion of 0-999 integer value into a given buffer
- warning: it won't check that Value is in 0-999 range
- up to 4 bytes may be written to the buffer (including #0 terminator)
procedure AppendBuffersToRawUtf8(var Text: RawUtf8; const Buffers: array of PUtf8Char);
Fast add some characters to a RawUtf8
string
- faster than Text
:= Text
+RawUtf8
(Buffers[0])+RawUtf8
(Buffers[0])+...
- will handle up to 64 Buffers[] - raise an ESynException
on too many Buffers
function AppendBufferToBuffer(Buffer: PUtf8Char; Text: pointer; Len: PtrInt): PUtf8Char;
Fast add some characters from ane buffer into another buffer
- warning: the Buffer should contain enough space to store
the Text
, otherwise you may encounter buffer overflows and random memory errors
procedure AppendBufferToRawByteString(var Content: RawByteString; const Buffer; BufferLen: PtrInt); overload;
Just redirect to mormot.core.text Append
(...) overloads
procedure AppendCharOnceToRawUtf8(var Text: RawUtf8; Ch: AnsiChar);
Fast add one character to a RawUtf8
string, if not already present
- avoid a temporary memory allocation of a string, so faster alternative to
if (Text<>'') and (Text[length(Text)]<>Ch) then Text := Text + ch;
function AppendRawUtf8ToBuffer(Buffer: PUtf8Char; const Text: RawUtf8): PUtf8Char;
Fast add some characters from a RawUtf8
string into a given buffer
- warning: the Buffer should contain enough space to store
the Text
, otherwise you may encounter buffer overflows and random memory errors
function AppendUInt32ToBuffer(Buffer: PUtf8Char; Value: PtrUInt): PUtf8Char;
Fast add text
conversion of a 32-bit signed integer value into a given buffer
- warning: the Buffer should contain enough space to store
the text
, otherwise you may encounter buffer overflows and random memory errors
function AreUrlValid(const Url: array of RawUtf8): boolean;
Checks if the supplied UTF-8 text
values don't need URI encoding
- returns TRUE if all its chars of all strings are non-void plain ASCII-7 RFC compatible identifiers (0..9a..zA..Z-_.~)
function AsciiToBaudot(P: PAnsiChar; len: PtrInt): RawByteString; overload;
Convert some ASCII-7 text
into binary, using Emile Baudot code
- as used in telegraphs, covering #10 #13 #32 a-z 0-9 - ' , ! : ( + ) $ ? @ . / ; charset, following a custom static-huffman-like encoding with 5-bit masks
- any upper case char will be converted into lowercase
during encoding
- other characters (e.g. UTF-8 accents, or controls chars) will be ignored
- resulting binary will consume 5 (or 10) bits per character
- reverse
of the BaudotToAscii
() function
- the "baud" symbol rate measurement comes from Emile's name ;)
function AsciiToBaudot(const Text: RawUtf8): RawByteString; overload;
Convert some ASCII-7 text
into binary, using Emile Baudot code
- as used in telegraphs, covering #10 #13 #32 a-z 0-9 - ' , ! : ( + ) $ ? @ . / ; charset, following a custom static-huffman-like encoding with 5-bit masks
- any upper case char will be converted into lowercase
during encoding
- other characters (e.g. UTF-8 accents, or controls chars) will be ignored
- resulting binary will consume 5 (or 10) bits per character
- reverse
of the BaudotToAscii
() function
- the "baud" symbol rate measurement comes from Emile's name ;)
function Base32ToBin(B32: PAnsiChar; B32Len: integer): RawByteString; overload;
Conversion from Base32 encoded text
into a binary string
- RFC4648 Base32 is defined as upper alphanumeric without misleading 0O 1I 8B
- returns '' if input was not valid Base32 encoded
function Base32ToBin(const base32: RawUtf8): RawByteString; overload;
Conversion from Base32 encoded text
into a binary string
- RFC4648 Base32 is defined as upper alphanumeric without misleading 0O 1I 8B
- returns '' if input was not valid Base32 encoded
function Base58ToBin(B58: PAnsiChar; B58Len: integer; var Dest: TSynTempBuffer): integer; overload;
Conversion from Base58 encoded text
into a binary buffer
- Bitcoin' Base58 was defined as alphanumeric chars without misleading 0O I1
- Base58 is much slower than Base64, performing in O(n^2) instead of O(n), and should not be used on big buffers
- returns the number of decoded chars encoded into Dest.buf
- caller should call Dest.Done once it is finished with the output binary
function Base58ToBin(B58: PAnsiChar; B58Len: integer): RawByteString; overload;
Conversion from Base58 encoded text
into a binary string
- Bitcoin' Base58 was defined as alphanumeric chars without misleading 0O I1
- Base58 is much slower than Base64, and should not be used on big buffers
- returns '' if input was not valid Base58 encoded
function Base58ToBin(const base58: RawUtf8): RawByteString; overload;
Conversion from Base58 encoded text
into a binary string
- Bitcoin' Base58 was defined as alphanumeric chars without misleading 0O I1
- Base58 is much slower than Base64, and should not be used on big buffers
- returns '' if input was not valid Base58 encoded
function Base64Decode(sp, rp: PAnsiChar; len: PtrInt): boolean;
Direct low-level decoding of a Base64 encoded buffer
- here len is the number of 4 chars chunks in sp input
- deprecated low-level function: use Base64ToBin
/Base64ToBinSafe
instead
procedure Base64Encode(rp, sp: PAnsiChar; len: cardinal);
Raw function for efficient binary to Base64 encoding
- just a wrapper around Base64EncodeMain
() + Base64EncodeTrailing
()
procedure Base64EncodeTrailing(rp, sp: PAnsiChar; len: cardinal);
Raw function for efficient binary to Base64 encoding of the last bytes
- don't use this function, but rather the BinToBase64
() overloaded functions
function Base64MagicCheckAndDecode(Value: PUtf8Char; ValueLen: integer; var Blob: RawByteString): boolean; overload;
Check and decode '\uFFF0base64encodedbinary' content into binary
- this method will check the supplied value to match the expected JSON_BASE64_MAGIC_C
pattern, decode and set Blob and return TRUE
function Base64MagicCheckAndDecode(Value: PUtf8Char; var Blob: TSynTempBuffer; ValueLen: integer = 0): boolean; overload;
Check and decode '\uFFF0base64encodedbinary' content into binary
- this method will check the supplied value to match the expected JSON_BASE64_MAGIC_C
pattern, decode and set Blob and return TRUE
function Base64MagicCheckAndDecode(Value: PUtf8Char; var Blob: RawByteString): boolean; overload;
Check and decode '\uFFF0base64encodedbinary' content into binary
- this method will check the supplied value to match the expected JSON_BASE64_MAGIC_C
pattern, decode and set Blob and return TRUE
procedure Base64MagicDecode(var ParamValue: RawUtf8);
Just a wrapper around Base64ToBin
() for in-place decode of JSON_BASE64_MAGIC_C
'\uFFF0base64encodedbinary' content into binary
- input ParamValue shall have been checked to match the expected pattern
procedure Base64MagicToBlob(Base64: PUtf8Char; var result: RawUtf8);
Convert a Base64-encoded content into binary hexadecimal ready for SQL
- returns e.g. X'53514C697465'
function Base64MagicTryAndDecode(Value: PUtf8Char; ValueLen: integer; var Blob: RawByteString): boolean;
Decode '\uFFF0base64encodedbinary' or 'base64encodedbinary' into binary
- same as Base64MagicCheckAndDecode
(), but will detect and ignore the magic and not require it
function Base64ToBin(base64, bin: PAnsiChar; base64len, binlen: PtrInt ; nofullcheck: boolean = true ): boolean; overload;
Fast conversion from Base64 encoded text
into binary data
- returns TRUE on success, FALSE if base64 does not match binlen
- nofullcheck is deprecated and not used any more, since nofullcheck=false is now processed with no performance cost
function Base64ToBin(const base64: RawByteString; bin: PAnsiChar; binlen: PtrInt ; nofullcheck: boolean = true ): boolean; overload;
Fast conversion from Base64 encoded text
into binary data
- returns TRUE on success, FALSE if base64 does not match binlen
- nofullcheck is deprecated and not used any more, since nofullcheck=false is now processed with no performance cost
function Base64ToBin(sp: PAnsiChar; len: PtrInt; var Blob: TSynTempBuffer): boolean; overload;
Fast conversion from Base64 encoded text
into binary data
- returns TRUE on success, FALSE if sp/len buffer was invvalid
function Base64ToBin(const s: RawByteString): RawByteString; overload;
Fast conversion from Base64 encoded text
into binary data
- is now just an alias to Base64ToBinSafe
() overloaded function
- returns '' if s was not a valid Base64-encoded input
function Base64ToBin(sp: PAnsiChar; len: PtrInt): RawByteString; overload;
Fast conversion from Base64 encoded text
into binary data
- is now just an alias to Base64ToBinSafe
() overloaded function
- returns '' if sp/len buffer was not a valid Base64-encoded input
function Base64ToBin(sp: PAnsiChar; len: PtrInt; var data: RawByteString): boolean; overload;
Fast conversion from Base64 encoded text
into binary data
- is now just an alias to Base64ToBinSafe
() overloaded function
- returns false and data='' if sp/len buffer was invalid
function Base64ToBinLength(sp: PAnsiChar; len: PtrInt): PtrInt;
Retrieve the expected undecoded length of a Base64 encoded buffer
- here len is the number of bytes in sp
function Base64ToBinLengthSafe(sp: PAnsiChar; len: PtrInt): PtrInt;
Retrieve the expected undecoded length of a Base64 encoded buffer
- here len is the number of bytes in sp
- will check supplied text
is a valid Base64 encoded stream
function Base64ToBinSafe(const s: RawByteString): RawByteString; overload;
Fast conversion from Base64 encoded text
into binary data
- will check supplied text
is a valid Base64 encoded stream
function Base64ToBinSafe(sp: PAnsiChar; len: PtrInt): RawByteString; overload;
Fast conversion from Base64 encoded text
into binary data
- will check supplied text
is a valid Base64 encoded stream
function Base64ToBinSafe(sp: PAnsiChar; len: PtrInt; var data: RawByteString): boolean; overload;
Fast conversion from Base64 encoded text
into binary data
- will check supplied text
is a valid Base64 encoded stream
function Base64ToBinSafe(sp: PAnsiChar; len: PtrInt; out data: TBytes): boolean; overload;
Fast conversion from Base64 encoded text
into binary data
- will check supplied text
is a valid Base64 encoded stream
function Base64ToBinTrim(const s: RawByteString): RawByteString;
Conversion from Base64 encoded text
into binary data, ignoring spaces
- returns '' if s was not a valid Base64-encoded input once spaces are trimmed
- consider PemToDer
() from mormot.crypt.secure if you need to read PEM content
procedure Base64ToUri(var base64: RawUtf8);
Conversion from any Base64 encoded value into URI-compatible encoded text
- warning: will modify the supplied base64 string in-place
- in comparison to Base64 standard encoding, will trim
any right-sided '=' unsignificant characters, and replace '+' or '/' by '_' or '-'
function Base64uriDecode(sp, rp: PAnsiChar; len: PtrInt): boolean;
Direct low-level decoding of a Base64-URI encoded buffer
- the buffer is expected to be at least Base64uriToBinLength
() bytes long
- returns true if the supplied sp[] buffer has been successfully decoded into rp[] - will break at any invalid character, so is always safe to use
- in comparison to Base64 standard encoding, will trim
any right-sided '=' unsignificant characters, and replace '+' or '/' by '_' or '-'
- you should better not use this, but Base64uriToBin
() overloaded functions
procedure Base64uriEncode(rp, sp: PAnsiChar; len: cardinal);
Low-level conversion from a binary buffer into Base64-like URI-compatible encoded text
- you should rather use the overloaded BinToBase64uri
() functions
function Base64uriToBin(sp: PAnsiChar; len: PtrInt; var temp: TSynTempBuffer): boolean; overload;
Fast conversion from Base64-URI encoded text
into binary data
- caller should always execute temp.Done when finished with the data
- in comparison to Base64 standard encoding, will trim
any right-sided '=' unsignificant characters, and replace '+' or '/' by '_' or '-'
function Base64uriToBin(sp: PAnsiChar; len: PtrInt; var bin: RawByteString): boolean; overload;
Fast conversion from Base64-URI encoded text
into binary data
- in comparison to Base64 standard encoding, will trim
any right-sided '=' unsignificant characters, and replace '+' or '/' by '_' or '-'
function Base64uriToBin(const s: RawByteString): RawByteString; overload;
Fast conversion from Base64-URI encoded text
into binary data
- in comparison to Base64 standard encoding, will trim
any right-sided '=' unsignificant characters, and replace '+' or '/' by '_' or '-'
function Base64uriToBin(const base64: RawByteString; bin: PAnsiChar; binlen: PtrInt): boolean; overload;
Fast conversion from Base64-URI encoded text
into binary data
- in comparison to Base64 standard encoding, will trim
any right-sided '=' unsignificant characters, and replace '+' or '/' by '_' or '-'
- will check supplied text
is a valid Base64-URI encoded stream
function Base64uriToBin(base64, bin: PAnsiChar; base64len, binlen: PtrInt): boolean; overload;
Fast conversion from Base64-URI encoded text
into binary data
- in comparison to Base64 standard encoding, will trim
any right-sided '=' unsignificant characters, and replace '+' or '/' by '_' or '-'
- will check supplied text
is a valid Base64-URI encoded stream
function Base64uriToBin(sp: PAnsiChar; len: PtrInt): RawByteString; overload;
Fast conversion from Base64-URI encoded text
into binary data
- in comparison to Base64 standard encoding, will trim
any right-sided '=' unsignificant characters, and replace '+' or '/' by '_' or '-'
function Base64uriToBinLength(len: PtrInt): PtrInt;
Retrieve the expected undecoded length of a Base64-URI encoded buffer
- here len is the number of bytes in sp
- in comparison to Base64 standard encoding, will trim
any right-sided '=' unsignificant characters, and replace '+' or '/' by '_' or '-'
function BaudotToAscii(const Baudot: RawByteString): RawUtf8; overload;
Convert some Baudot code binary, into ASCII-7 text
- reverse
of the AsciiToBaudot
() function
- any uppercase
character would be decoded as lowercase
- and some characters may have disapeared outside of a-z 0-9 - ' , ! : ( + ) $ ? @ . / ; range
- the "baud" symbol rate measurement comes from Emile's name ;)
function BaudotToAscii(Baudot: PByteArray; len: PtrInt): RawUtf8; overload;
Convert some Baudot code binary, into ASCII-7 text
- reverse
of the AsciiToBaudot
() function
- any uppercase
character would be decoded as lowercase
- and some characters may have disapeared outside of a-z 0-9 - ' , ! : ( + ) $ ? @ . / ; range
- the "baud" symbol rate measurement comes from Emile's name ;)
procedure BinToBase32(Bin: PByteArray; Dest: PAnsiChar; BinLen: PtrInt; b32enc: PAnsiChar); overload;
Conversion from a binary buffer into Base32 encoded text
buffer
- default b32enc is RFC4648 upper alphanumeric without misleading 0O 1I 8B
function BinToBase32(Bin: PAnsiChar; BinLen: PtrInt): RawUtf8; overload;
Conversion from a binary buffer into Base32 encoded text
as RawUtf8
- RFC4648 Base32 is defined as upper alphanumeric without misleading 0O 1I 8B
function BinToBase32(const Bin: RawByteString): RawUtf8; overload;
Conversion from a binary buffer into Base32 encoded text
as RawUtf8
- RFC4648 Base32 is defined as upper alphanumeric without misleading 0O 1I 8B
function BinToBase32Length(BinLen: cardinal): cardinal;
Compute the length resulting of Base32 encoding of a binary buffer
- RFC4648 Base32 is defined as upper alphanumeric without misleading 0O 1I 8B
function BinToBase58(const Bin: RawByteString): RawUtf8; overload;
Conversion from a binary buffer into Base58 encoded text
as RawUtf8
- Bitcoin' Base58 was defined as alphanumeric chars without misleading 0O I1
- Base58 is much slower than Base64, performing in O(n^2) instead of O(n), and should not be used on big buffers
function BinToBase58(Bin: PAnsiChar; BinLen: integer): RawUtf8; overload;
Conversion from a binary buffer into Base58 encoded text
as RawUtf8
- Bitcoin' Base58 was defined as alphanumeric chars without misleading 0O I1
- Base58 is much slower than Base64, performing in O(n^2) instead of O(n), and should not be used on big buffers
function BinToBase58(Bin: PAnsiChar; BinLen: integer; var Dest: TSynTempBuffer): integer; overload;
Conversion from a binary buffer into Base58 encoded text
as TSynTempBuffer
- Bitcoin' Base58 was defined as alphanumeric chars without misleading 0O I1
- Base58 is much slower than Base64, performing in O(n^2) instead of O(n), and should not be used on big buffers
- returns the number of encoded chars encoded into Dest.buf
- caller should call Dest.Done once it is finished with the output text
function BinToBase64(Bin: PAnsiChar; BinBytes: integer): RawUtf8; overload;
Fast conversion from binary data into Base64 encoded UTF-8 text
function BinToBase64(const s: RawByteString): RawUtf8; overload;
Fast conversion from binary data into Base64 encoded UTF-8 text
function BinToBase64(const data, Prefix, Suffix: RawByteString; WithMagic: boolean): RawUtf8; overload;
Fast conversion from binary data into prefixed/suffixed Base64 encoded UTF-8 text
- with optional JSON_BASE64_MAGIC_C
prefix (UTF-8 encoded \uFFF0 special code)
- may use AVX2 optimized asm (10GB/s) on FPC x86_64
function BinToBase64Length(len: PtrUInt): PtrUInt;
Retrieve the expected encoded length after Base64 process
function BinToBase64Line(sp: PAnsiChar; len: PtrUInt; const Prefix: RawUtf8 = ''; const Suffix: RawUtf8 = ''): RawUtf8;
Fast conversion from binary into prefixed/suffixed Base64 with 64 chars per line
function BinToBase64Short(Bin: PAnsiChar; BinBytes: integer): ShortString; overload;
Fast conversion from a small binary data into Base64 encoded UTF-8 text
function BinToBase64Short(const s: RawByteString): ShortString; overload;
Fast conversion from a small binary data into Base64 encoded UTF-8 text
function BinToBase64uri(Bin: PAnsiChar; BinBytes: integer): RawUtf8; overload;
Fast conversion from a binary buffer into Base64-like URI-compatible encoded text
- in comparison to Base64 standard encoding, will trim
any right-sided '=' unsignificant characters, and replace '+' or '/' by '_' or '-'
function BinToBase64uri(const s: RawByteString): RawUtf8; overload;
Fast conversion from binary data into Base64-like URI-compatible encoded text
- in comparison to Base64 standard encoding, will trim
any right-sided '=' unsignificant characters, and replace '+' or '/' by '_' or '-'
function BinToBase64uriLength(len: PtrUInt): PtrUInt;
Retrieve the expected encoded length after Base64-URI process
- in comparison to Base64 standard encoding, will trim
any right-sided '=' unsignificant characters, and replace '+' or '/' by '_' or '-'
function BinToBase64uriShort(Bin: PAnsiChar; BinBytes: integer): ShortString;
Fast conversion from a binary buffer into Base64-like URI-compatible encoded ShortString
- in comparison to Base64 standard encoding, will trim
any right-sided '=' unsignificant characters, and replace '+' or '/' by '_' or '-'
- returns '' if BinBytes void or too big for the resulting ShortString
function BinToBase64WithMagic(const data: RawByteString): RawUtf8; overload;
Fast conversion from binary data into Base64 encoded UTF-8 text
with JSON_BASE64_MAGIC_C
prefix (UTF-8 encoded \uFFF0 special code)
procedure BinToBase64WithMagic(Data: pointer; DataLen: integer; var Result: RawUtf8); overload;
Fast conversion from binary data into Base64 encoded UTF-8 text
with JSON_BASE64_MAGIC_C
prefix (UTF-8 encoded \uFFF0 special code)
function BinToBase64WithMagic(Data: pointer; DataLen: integer): RawUtf8; overload;
Fast conversion from binary data into Base64 encoded UTF-8 text
with JSON_BASE64_MAGIC_C
prefix (UTF-8 encoded \uFFF0 special code)
function BinToHumanHex(Data: PByte; Len: integer; PerLine: integer = 16; LeftTab: integer = 0; SepChar: AnsiChar = ':'): RawUtf8; overload;
Generate some 'xx:xx:xx:xx' output buffer with left and right margins
- used e.g. by ParsedToText
() to output X509
public key content in PeerInfo
procedure BinToHumanHex(W: TTextWriter; Data: PByte; Len: integer; PerLine: integer = 16; LeftTab: integer = 0; SepChar: AnsiChar = ':'); overload;
Generate some 'xx:xx:xx:xx' output buffer with left and right margins
function BinToSource(const ConstName, Comment: RawUtf8; const Data: RawByteString; PerLine: integer = 16; const Suffix: RawUtf8 = ''): RawUtf8; overload;
Generate some pascal source code holding some data binary as constant
function BinToSource(const ConstName, Comment: RawUtf8; Data: pointer; Len: integer; PerLine: integer = 16; const Suffix: RawUtf8 = ''): RawUtf8; overload;
Generate some pascal source code holding some data binary as constant
- can store
sensitive information (e.g. certificates) within the executable
- generates a source code snippet of the following format:
const // Comment ConstName: array[0..2] of byte = ( $01, $02, $03);
procedure BinToSource(Dest: TTextWriter; const ConstName, Comment: RawUtf8; Data: pointer; Len: integer; PerLine: integer = 16); overload;
Generate some pascal source code holding some data binary as constant
- can store
sensitive information (e.g. certificates) within the executable
- generates a source code snippet of the following format:
const // Comment ConstName: array[0..2] of byte = ( $01, $02, $03);
function BlobToBytes(P: PUtf8Char): TBytes;
Create a TBytes
from TEXT
-encoded blob data
- blob data can be encoded as SQLite3 BLOB literals (X'53514C697465' e.g.) or or Base64 encoded content ('\uFFF0base64encodedbinary') or plain TEXT
procedure BlobToRawBlob(P: PUtf8Char; var result: RawBlob; Len: integer = 0); overload;
Fill a RawBlob
from TEXT
-encoded blob data
- blob data can be encoded as SQLite3 BLOB literals (X'53514C697465' e.g.) or or Base64 encoded content ('\uFFF0base64encodedbinary') or plain TEXT
function BlobToRawBlob(const Blob: RawByteString): RawBlob; overload;
Fill a RawBlob
from TEXT
-encoded blob data
- blob data can be encoded as SQLite3 BLOB literals (X'53514C697465' e.g.) or or Base64 encoded content ('\uFFF0base64encodedbinary') or plain TEXT
function BlobToRawBlob(P: PUtf8Char; Len: integer = 0): RawBlob; overload;
Fill a RawBlob
from TEXT
-encoded blob data
- blob data can be encoded as SQLite3 BLOB literals (X'53514C697465' e.g.) or or Base64 encoded content ('\uFFF0base64encodedbinary') or plain TEXT
function BlobToStream(P: PUtf8Char): TStream;
Create a memory stream from TEXT
-encoded blob data
- blob data can be encoded as SQLite3 BLOB literals (X'53514C697465' e.g.) or or Base64 encoded content ('\uFFF0base64encodedbinary') or plain TEXT
- the caller must free the stream instance after use
procedure BytesToRawByteString(const bytes: TBytes; out buf: RawByteString);
Creates a RawByteString
memory buffer from a TBytes
content
function ContentToShort(const source: RawByteString): ShortString;
If source is not UTF-8 calls EscapeToShort
, otherwise return it directly
function EmojiFromDots(const text: RawUtf8): RawUtf8; overload;
Conversion of github/Markdown :identifiers: into UTF-8 Emoji sequences
procedure EmojiFromDots(P: PUtf8Char; W: TTextWriter); overload;
Low-level conversion of github/Markdown :identifiers: into UTF-8 Emoji sequences
function EmojiFromText(P: PUtf8Char; len: PtrInt): TEmoji;
Recognize github/Markdown compatible text
of Emojis
- for instance 'sunglasses' text
buffer will return eSunglasses
- returns eNone if no case-insensitive match was found
function EmojiParseDots(var P: PUtf8Char; W: TTextWriter = nil): TEmoji;
Low-level parser of github/Markdown compatible text
of Emojis
- supplied P^ should point to ':'
- will append
the recognized UTF-8 Emoji if P contains e.g. :joy: or :)
- will append
':' if no Emoji text
is recognized, and return eNone
- will try both EMOJI_AFTERDOTS
[] and EMOJI_RTTI
[] reference set
- if W is nil, won't append
anything, but just return the recognized TEmoji
function EmojiToDots(const text: RawUtf8): RawUtf8; overload;
Conversion of UTF-8 Emoji sequences into github/Markdown :identifiers:
procedure EmojiToDots(P: PUtf8Char; W: TTextWriter); overload;
Low-level conversion of UTF-8 Emoji sequences into github/Markdown :identifiers:
function EscapeBuffer(s: PAnsiChar; slen: integer; d: PAnsiChar; dmax: integer): PAnsiChar;
Low-level fast conversion from binary data to escaped text
- non printable characters will be written as $xx hexadecimal codes
- will be #0 terminated, with '...' characters trailing on dmax overflow
- ensure the destination buffer contains at least dmax bytes, which is always the case when using LogEscape
() and its local TLogEscape
variable
function EscapeToShort(source: PAnsiChar; sourcelen: integer): ShortString; overload;
Fill a ShortString with the (hexadecimal) chars of the input text
/binary
function EscapeToShort(const source: RawByteString): ShortString; overload;
Fill a ShortString with the (hexadecimal) chars of the input text
/binary
function FromVarBlob(Data: PByte): TValueResult;
Retrieve pointer and length to a variable-length text
/blob buffer
function FromVarInt32(var Source: PByte): integer;
Convert a 32-bit variable-length integer buffer into an integer
- decode negative values from cardinal two-complement, i.e. 0=0,1=1,2=-1,3=2,4=-2...
function FromVarInt64(var Source: PByte): Int64;
Convert a 64-bit variable-length integer buffer into a Int64
function FromVarInt64Value(Source: PByte): Int64;
Convert a 64-bit variable-length integer buffer into a Int64
- this version won't update the Source pointer
procedure FromVarString(var Source: PByte; var Value: RawByteString; CodePage: integer); overload;
Retrieve a variable-length text
buffer
- this overloaded function will set the supplied code page to the AnsiString
function FromVarString(var Source: PByte; SourceMax: PByte; var Value: RawByteString; CodePage: integer): boolean; overload;
Retrieve a variable-length text
buffer
- this overloaded function will set the supplied code page to the AnsiString and will also check for the SourceMax end of buffer
- returns TRUE on success, or FALSE on any buffer overload detection
procedure FromVarString(var Source: PByte; var Value: RawUtf8); overload;
Retrieve a variable-length UTF-8 encoded text
buffer in a newly allocation RawUtf8
function FromVarString(var Source: PByte; SourceMax: PByte): RawUtf8; overload;
Safe retrieve a variable-length UTF-8 encoded text
buffer in a newly allocation RawUtf8
- supplied SourceMax value will avoid any potential buffer overflow
function FromVarString(var Source: PByte): RawUtf8; overload;
Retrieve a variable-length UTF-8 encoded text
buffer in a newly allocation RawUtf8
function FromVarString(var Source: PByte; SourceMax: PByte; var Value: TSynTempBuffer): boolean; overload;
Retrieve a variable-length UTF-8 encoded text
buffer in a temporary buffer
- caller should call Value.Done after use of the Value.buf memory
- this overloaded function will also check for the SourceMax end of buffer, returning TRUE on success, or FALSE on any buffer overload detection
procedure FromVarString(var Source: PByte; var Value: TSynTempBuffer); overload;
Retrieve a variable-length UTF-8 encoded text
buffer in a temporary buffer
- caller should call Value.Done after use of the Value.buf memory
- this overloaded function would include a #0 terminator, so Value.buf could be parsed as a valid PUtf8Char
buffer (e.g. containing JSON)
function FromVarUInt32(var Source: PByte; SourceMax: PByte; out Value: cardinal): boolean; overload;
Convert a 32-bit variable-length integer buffer into a cardinal
- will call FromVarUInt32
() if SourceMax=nil, or FromVarUInt32Safe
() if set
- returns false on error, true if Value has been set properly
function FromVarUInt32(var Source: PByte): cardinal; overload;
Convert a 32-bit variable-length integer buffer into a cardinal
- fast inlined process for any number < 128
- use overloaded FromVarUInt32
() or FromVarUInt32Safe
() with a SourceMax pointer to avoid any potential buffer overflow
- use FromVarUInt32Big
() is the content is likely to be >= 128
function FromVarUInt32Big(var Source: PByte): cardinal;
Convert a 32-bit variable-length integer buffer into a cardinal
- this version could be called if number is likely to be > $7f, so if inlining the first byte won't make
any benefit
function FromVarUInt32High(var Source: PByte): cardinal;
Convert a 32-bit variable-length integer buffer into a cardinal
- this version must be called if Source^ has already been checked to be > $7f
function FromVarUInt32Safe(Source, SourceMax: PByte; out Value: cardinal): PByte;
Safely convert a 32-bit variable-length integer buffer into a cardinal
- slower but safer process checking out of boundaries memory access in Source
- SourceMax is expected to be not nil, and to point to the first byte just after the Source memory buffer
- returns nil on error, or point to next input data on successful decoding
function FromVarUInt32Up128(var Source: PByte): cardinal;
Convert a 32-bit variable-length integer buffer into a cardinal
- used e.g. when inlining FromVarUInt32
()
- this version must be called if Source^ has already been checked to be > $7f
result := Source^; inc(Source); if result>$7f then result := (result and $7F) or FromVarUInt32Up128(Source);
function FromVarUInt64(var Source: PByte; SourceMax: PByte; out Value: Qword): boolean; overload;
Convert a 64-bit variable-length integer buffer into a UInt64
- will call FromVarUInt64
() if SourceMax=nil, or FromVarUInt64Safe
() if set
- returns false on error, true if Value has been set properly
function FromVarUInt64(var Source: PByte): QWord; overload;
Convert a 64-bit variable-length integer buffer into a UInt64
function FromVarUInt64Safe(Source, SourceMax: PByte; out Value: QWord): PByte;
Safely convert a 64-bit variable-length integer buffer into a UInt64
- slower but safer process checking out of boundaries memory access in Source
- SourceMax is expected to be not nil, and to point to the first byte just after the Source memory buffer
- returns nil on error, or point to next input data on successful decoding
function GetJpegSize(jpeg: PAnsiChar; len: PtrInt; out Height, Width, Bits: integer): boolean; overload;
Fast guess of the size, in pixels, of a JPEG memory buffer
- will only scan for basic JPEG structure, up to the StartOfFrame (SOF) chunk
- returns TRUE if the buffer is likely to be a JPEG picture, and set the Height + Width + Bits variable with its dimensions - but there may be false positive recognition, and no waranty that the memory buffer is a valid JPEG
- returns FALSE if the buffer does not have any expected SOI/SOF markers
function GetMimeContentType(Content: pointer; Len: PtrInt; const FileName: TFileName = ''; const DefaultContentType: RawUtf8 = BINARY_CONTENT_TYPE; Mime: PMimeType = nil): RawUtf8;
Retrieve the MIME content type from its file name or a supplied binary buffer
- will first check for known file extensions, then inspect the binary content
- return the MIME type, ready to be appended to a 'Content-Type: ' HTTP
header
- default is DefaultContentType or 'application/octet-stream' (BINARY_CONTENT_TYPE
) or 'application/fileextension' if FileName was specified
- see @http
://en.wikipedia.org/wiki/Internet_media_type for most common values
function GetMimeContentTypeFromBuffer(Content: pointer; Len: PtrInt; const DefaultContentType: RawUtf8; Mime: PMimeType = nil): RawUtf8;
Retrieve the MIME content type from a supplied binary buffer
- inspect the first bytes, to guess from standard known headers
- return the MIME type, ready to be appended to a 'Content-Type: ' HTTP
header
- returns DefaultContentType if the binary buffer has an unknown layout
function GetMimeContentTypeFromExt(const FileName: TFileName; FileExt: PRawUtf8 = nil): TMimeType;
MtUnknown mtPng mtGif mtTiff mtJpg mtBmp mtDoc mtPpt mtXls mtHtml mtCss mtJS RFC 9239 mtXIcon mtFont RFC 8081 mtText mtSvg mtXml mtWebp mtManifest mtJson mtOgg RFC 5334 mtMp4 RFC 4337 6381 mtMp2 mtMpeg RFC 3003 mtH264 RFC 6184 mtWma mtWmv mtAvi mtGzip mtWebm mtRar mt7z mtZip mtBz2 mtPdf mtSQlite3 mtXcomp mtDicom retrieve the MIME content type from its file name
function GetMimeContentTypeFromMemory(Content: pointer; Len: PtrInt): TMimeType;
Retrieve the MIME content type from a supplied binary buffer
function GetMimeContentTypeHeader(const Content: RawByteString; const FileName: TFileName = ''; Mime: PMimeType = nil): RawUtf8;
Retrieve the HTTP
header for MIME content type from a supplied binary buffer
- just append
HEADER_CONTENT_TYPE
and GetMimeContentType
() result
- can be used as such:
Call.OutHead := GetMimeContentTypeHeader(Call.OutBody,aFileName);
function GetMimeTypeFromExt(const Ext: RawUtf8): TMimeType;
Retrieve the MIME content type from its file extension text
(without '.')
function GetStreamBuffer(S: TStream): pointer;
Retrieve the memory buffer of a TCustomMemoryStream/TRawByteStringStream
- returns nil if the instance is not of those classes
function GotoNextVarInt(Source: PByte): pointer;
Jump a value in the 32-bit or 64-bit variable-length integer buffer
function GotoNextVarString(Source: PByte): pointer;
Jump a value in variable-length text
buffer
function HashFile(const FileName: TFileName; Hasher: THasher = nil): cardinal; overload;
Compute the 32-bit default hash
of a file content
- you can specify your own hashing function if DefaultHasher
is not what you expect
function HashFileCrc32c(const FileName: TFileName): RawUtf8;
Compute the crc32c
checksum of a given file
- this function maps the THashFile
signature
function HtmlEscapeMarkdown(const md: RawUtf8; esc: TTextWriterHtmlEscape = [heEmojiToUtf8]): RawUtf8;
Escape some Markdown-marked text
into HTML
- just a wrapper around AddHtmlEscapeMarkdown
() process
function HtmlEscapeWiki(const wiki: RawUtf8; esc: TTextWriterHtmlEscape = [heHtmlEscape, heEmojiToUtf8]): RawUtf8;
Escape some wiki-marked text
into HTML
- just a wrapper around AddHtmlEscapeWiki
() process
function IncludeTrailingUriDelimiter(const URI: RawByteString): RawByteString;
Ensure the supplied URI contains a trailing '/' charater
function IsBase64(const s: RawByteString): boolean; overload;
Check if the supplied text
is a valid Base64 encoded stream
function IsBase64(sp: PAnsiChar; len: PtrInt): boolean; overload;
Check if the supplied text
is a valid Base64 encoded stream
function isBlobHex(P: PUtf8Char): boolean;
Return true if the TEXT
is encoded as SQLite3 BLOB literals (X'53514C697465' e.g.)
function IsContentCompressed(Content: pointer; Len: PtrInt): boolean;
Retrieve if some content is compressed, from a supplied binary buffer
- returns TRUE, if the header in binary buffer "may" be compressed (this method can trigger false positives), e.g. begin with most common already compressed zip/gz/gif/png/jpeg/avi/mp3/mp4 markers (aka "magic numbers")
function IsContentTypeCompressible(ContentType: PUtf8Char): boolean;
Recognize e.g. 'text
/css' or 'application/json' as compressible
function IsContentTypeCompressibleU(const ContentType: RawUtf8): boolean;
Recognize e.g. 'text
/css' or 'application/json' as compressible
function IsContentTypeJson(ContentType: PUtf8Char): boolean;
Recognize e.g. 'application/json' or 'application/vnd.api+json'
function IsStreamBuffer(S: TStream): boolean;
Check if class is a TCustomMemoryStream/TRawByteStringStream
function IsUrlValid(P: PUtf8Char): boolean;
Checks if the supplied UTF-8 text
don't need URI encoding
- returns TRUE if all its chars are non-void plain ASCII-7 RFC compatible identifiers (0..9a..zA..Z-_.~)
function LogEscape(source: PAnsiChar; sourcelen: integer; var temp: TLogEscape; enabled: boolean = true): PAnsiChar;
Fill TLogEscape
stack buffer with the (hexadecimal) chars of the input binary
- up to 512 bytes will be escaped and appended to a local temp: TLogEscape
variable, using the EscapeBuffer
() low-level function
- you can then log the resulting escaped text
by passing the returned PAnsiChar as % parameter to a TSynLog.Log
() method
- the "enabled" parameter can be assigned from a process option, avoiding to process the escape if verbose logs are disabled
- used e.g. to implement logBinaryFrameContent option for WebSockets
function LogEscapeFull(const source: RawByteString): RawUtf8; overload;
Returns a text
buffer with the (hexadecimal) chars of the input binary
- is much slower than LogEscape
/EscapeToShort
, but has no size limitation
function LogEscapeFull(source: PAnsiChar; sourcelen: integer): RawUtf8; overload;
Returns a text
buffer with the (hexadecimal) chars of the input binary
- is much slower than LogEscape
/EscapeToShort
, but has no size limitation
function MultiPartFormDataAddField(const FieldName, FieldValue: RawUtf8; var MultiPart: TMultiPartDynArray; const ForcedContentType: RawUtf8 = ''): boolean;
Encode a field in a multipart array
- FieldName: field name of the part
- FieldValue: value of the field
- Multipart: where the part is added
- consider THttpMultiPartStream
from mormot.net.client for huge file content
function MultiPartFormDataAddFile(const FileName: TFileName; var MultiPart: TMultiPartDynArray; const Name: RawUtf8 = ''; const ForcedContentType: RawUtf8 = ''): boolean;
Encode a file in a multipart array
- FileName: file to encode
- Multipart: where the part is added
- Name: name of the part, is empty the name 'File###' is generated
- consider THttpMultiPartStream
from mormot.net.client for huge file content
function MultiPartFormDataDecode(const MimeType, Body: RawUtf8; var MultiPart: TMultiPartDynArray): boolean;
Decode multipart/form-data POST request content into memory
- following RFC 1867
- decoded sections are appended to MultiPart[] existing array
function MultiPartFormDataEncode(const MultiPart: TMultiPartDynArray; var MultiPartContentType, MultiPartContent: RawUtf8; Rfc2388NestedFiles: boolean = false): boolean;
Encode multipart fields and files
- only one of them can be used because MultiPartFormDataDecode
must implement both decodings
- MultiPart: parts to build the multipart content from, which may be created using MultiPartFormDataAddFile
/MultiPartFormDataAddField
- MultiPartContentType: variable returning
Content-Type: multipart/form-data; boundary=xxx
where xxx is the first generated boundary
- MultiPartContent: generated multipart content
- Rfc2388NestedFiles will force the deprecated nested "multipart/mixed" format
- consider THttpMultiPartStream
from mormot.net.client for huge file content
function MultiPartFormDataNewBound(var boundaries: TRawUtf8DynArray): RawUtf8;
Used e.g. by MultiPartFormDataEncode
and THttpMultiPartStream.Add
function Plural(const itemname: ShortString; itemcount: cardinal): ShortString;
Write count number and append
's' (if needed) to form a plural
English noun
- for instance, Plural
('row',100) returns '100 rows' with no heap allocation
function RawBlobToBlob(const RawBlob: RawBlob): RawUtf8; overload;
Creates a TEXT
-encoded version of blob data from a RawBlob
- TEXT
will be encoded as SQLite3 BLOB literals (X'53514C697465' e.g.)
function RawBlobToBlob(RawBlob: pointer; RawBlobLength: integer): RawUtf8; overload;
Creates a TEXT
-encoded version of blob data from a memory data
- same as RawBlob
, but with direct memory access via a pointer/byte size pair
- TEXT
will be encoded as SQLite3 BLOB literals (X'53514C697465' e.g.)
function RawByteStringArrayConcat(const Values: array of RawByteString): RawByteString;
Fast concatenation of several AnsiStrings
procedure RawByteStringToBytes(const buf: RawByteString; out bytes: TBytes);
Creates a TBytes
from a RawByteString
memory buffer
procedure ResourceSynLZToRawByteString(const ResName: string; out buf: RawByteString; Instance: TLibHandle = 0);
Creates a RawByteString
memory buffer from an SynLZ-compressed embedded resource
- returns '' if the resource is not found
- this method would use SynLZDecompress
() after ResourceToRawByteString
(), with a ResType=PChar(10) (i.e. RC_DATA)
- you can specify a library (dll) resource instance handle, if needed
procedure ResourceToRawByteString(const ResName: string; ResType: PChar; out buf: RawByteString; Instance: TLibHandle = 0);
Creates a RawByteString
memory buffer from an embedded resource
- returns '' if the resource is not found
- warning: resources size may be rounded up to alignment
- you can specify a library (dll) resource instance handle, if needed
function SameFileContent(const One, Another: TFileName): boolean;
Compare two files by content, reading them by blocks
function ToVarInt32(Value: PtrInt; Dest: PByte): PByte;
Convert an integer into a 32-bit variable-length integer buffer
- store
negative values as cardinal two-complement, i.e. 0=0,1=1,2=-1,3=2,4=-2...
function ToVarInt64(Value: Int64; Dest: PByte): PByte;
Convert a Int64 into a 64-bit variable-length integer buffer
function ToVarString(const Value: RawUtf8; Dest: PByte): PByte;
Convert a RawUtf8
into an UTF-8 encoded variable-length buffer
function ToVarUInt32(Value: cardinal; Dest: PByte): PByte;
Convert a cardinal into a 32-bit variable-length integer buffer
function ToVarUInt32Length(Value: PtrUInt): PtrUInt;
Return the number of bytes necessary to store
a 32-bit variable-length integer
- i.e. the ToVarUInt32
() buffer size
function ToVarUInt32LengthWithData(Value: PtrUInt): PtrUInt;
Return the number of bytes necessary to store
some data with a its 32-bit variable-length integer length
function ToVarUInt64(Value: QWord; Dest: PByte): PByte;
Convert a UInt64 into a 64-bit variable-length integer buffer
function UrlAppend(const Uri: RawUtf8; const NameValuePairs: array of const; Options: TUrlEncoder = []): RawUtf8;
Append
some encoded Name,Value pairs parameters to an existing URI
- will check if Uri does already end with '?' or '&'
function UrlDecode(const s: RawUtf8): RawUtf8; overload;
Decode a UrlEncode
() URI encoded parameter into its original value
function UrlDecode(U: PUtf8Char): RawUtf8; overload;
Decode a UrlEncode
() URI encoded parameter into its original value
function UrlDecodeCardinal(U: PUtf8Char; const Upper: RawUtf8; var Value: cardinal; Next: PPUtf8Char = nil): boolean;
Decode a specified parameter compatible with URI encoding into its original cardinal numerical value
- UrlDecodeCardinal
('offset=20&where=LastName%3D%27M%C3%B4net%27','OFFSET=',O,@Next) will return Next^='where=...' and O=20
- Upper should be already uppercased, and end with a '=' character
- if Upper is not found, Value is not modified, and result is FALSE
- if Upper is found, Value is modified with the supplied content, and result is TRUE
function UrlDecodeDouble(U: PUtf8Char; const Upper: RawUtf8; var Value: double; Next: PPUtf8Char = nil): boolean;
Decode a specified parameter compatible with URI encoding into its original floating-point value
- UrlDecodeDouble
('price=20.45&where=LastName%3D%27M%C3%B4net%27','PRICE=',P,@Next) will return Next^='where=...' and P=20.45
- Upper should be already uppercased, and end with a '=' character
- if Upper is not found, Value is not modified, and result is FALSE
- if Upper is found, Value is modified with the supplied content, and result is TRUE
function UrlDecodeExtended(U: PUtf8Char; const Upper: RawUtf8; var Value: TSynExtended; Next: PPUtf8Char = nil): boolean;
Decode a specified parameter compatible with URI encoding into its original floating-point value
- UrlDecodeExtended
('price=20.45&where=LastName%3D%27M%C3%B4net%27','PRICE=',P,@Next) will return Next^='where=...' and P=20.45
- Upper should be already uppercased, and end with a '=' character
- if Upper is not found, Value is not modified, and result is FALSE
- if Upper is found, Value is modified with the supplied content, and result is TRUE
function UrlDecodeInt64(U: PUtf8Char; const Upper: RawUtf8; var Value: Int64; Next: PPUtf8Char = nil): boolean;
Decode a specified parameter compatible with URI encoding into its original Int64 numerical value
- UrlDecodeInt64
('offset=20&where=LastName%3D%27M%C3%B4net%27','OFFSET=',O,@Next) will return Next^='where=...' and O=20
- Upper should be already uppercased, and end with a '=' character
- if Upper is not found, Value is not modified, and result is FALSE
- if Upper is found, Value is modified with the supplied content, and result is TRUE
function UrlDecodeInteger(U: PUtf8Char; const Upper: RawUtf8; var Value: integer; Next: PPUtf8Char = nil): boolean;
Decode a specified parameter compatible with URI encoding into its original integer numerical value
- UrlDecodeInteger
('offset=20&where=LastName%3D%27M%C3%B4net%27','OFFSET=',O,@Next) will return Next^='where=...' and O=20
- Upper should be already uppercased, and end with a '=' character
- if Upper is not found, Value is not modified, and result is FALSE
- if Upper is found, Value is modified with the supplied content, and result is TRUE
function UrlDecodeName(U: PUtf8Char): RawUtf8; overload;
Decode a UrlEncodeName
() URI encoded network name into its original value
- only parameters - i.e. after '?' - should replace spaces by '+'
function UrlDecodeName(const s: RawUtf8): RawUtf8; overload;
Decode a UrlEncodeName
() URI encoded network name into its original value
- only parameters - i.e. after '?' - should replace spaces by '+'
function UrlDecodeNeedParameters(U, CsvNames: PUtf8Char): boolean;
Returns TRUE if all supplied parameters do exist in the URI encoded text
- CsvNames parameter shall provide as a CSV list of names
- e.g. UrlDecodeNeedParameters
('price=20.45&where=LastName%3D','price,where') will return TRUE
function UrlDecodeNextName(U: PUtf8Char; out Name: RawUtf8): PUtf8Char;
Decode a URI-encoded Name from an input buffer
- decoded value is set in Name out variable
- returns a pointer just after the decoded name, after the '='
- returns nil if there was no name=... pattern in U
function UrlDecodeNextNameValue(U: PUtf8Char; var Name, Value: RawUtf8): PUtf8Char;
Decode the next Name=Value&.... pair from input URI
- Name is returned directly (should be plain ASCII 7-bit text
)
- Value is returned after URI decoding (from %.. patterns)
- if a pair is decoded, return a PUtf8Char
pointer to the next pair in the input buffer, or points to #0 if all content has been processed
- if a pair is not decoded, return nil
function UrlDecodeNextValue(U: PUtf8Char; out Value: RawUtf8): PUtf8Char;
Decode a URI-encoded Value from an input buffer
- decoded value is set in Value out variable
- returns a pointer just after the decoded value (may points e.g. to
0 or '&') - it is up to the caller to continue the process or not
function UrlDecodeValue(U: PUtf8Char; const Upper: RawUtf8; var Value: RawUtf8; Next: PPUtf8Char = nil): boolean;
Decode a specified parameter compatible with URI encoding into its original textual value
- UrlDecodeValue
('select=%2A&where=LastName%3D%27M%C3%B4net%27','SELECT=',V,@Next) will return Next^='where=...' and V='*'
- Upper should be already uppercased, and end with a '=' character
- if Upper is not found, Value is not modified, and result is FALSE
- if Upper is found, Value is modified with the supplied content, and result is TRUE
procedure UrlDecodeVar(U: PUtf8Char; L: PtrInt; var result: RawUtf8; name: boolean);
Decode a UrlEncode
/UrlEncodeName
() URI encoded string into its original value
- name=false for parameters (after ?), to replace spaces by '+'
function UrlEncode(Text: PUtf8Char): RawUtf8; overload;
Encode a string as URI parameter encoding, i.e. ' ' as '+'
procedure UrlEncode(W: TTextWriter; Text: PUtf8Char; TextLen: PtrInt); overload;
Append
a string as URI parameter encoding, i.e. ' ' as '+'
procedure UrlEncode(W: TTextWriter; const Text: RawUtf8); overload;
Append
a string as URI parameter encoding, i.e. ' ' as '+'
function UrlEncode(const Text: RawUtf8): RawUtf8; overload;
Encode a string as URI parameter encoding, i.e. ' ' as '+'
function UrlEncode(const NameValuePairs: array of const; TrimLeadingQuestionMark: boolean): RawUtf8; overload;
Encode supplied parameters to be compatible with URI encoding
- consider using UrlAppend
() if you just need to append
some parameters
function UrlEncode(const NameValuePairs: array of const; Options: TUrlEncoder = []): RawUtf8; overload;
Encode supplied parameters to be compatible with URI encoding
- parameters must be supplied two by two, as Name,Value pairs, e.g.
url := UrlEncode(['select','*','where','ID=12','offset',23,'object',aObject]);
- parameters names should be plain ASCII-7 RFC compatible identifiers (0..9a..zA..Z_.~), otherwise they are skipped unless ueEncodeNames is set
- parameters values can be either textual, integer or extended, or any TObject
- TObject serialization into UTF-8 will be processed with ObjectToJson
()
function UrlEncodeFull(const PrefixFmt: RawUtf8; const PrefixArgs, NameValuePairs: array of const; Options: TUrlEncoder): RawUtf8;
Encode a full URI with prefix and parameters
procedure UrlEncodeName(W: TTextWriter; Text: PUtf8Char; TextLen: PtrInt); overload;
Append
a string as URI network name encoding, i.e. ' ' as %20
- only parameters - i.e. after '?' - should replace spaces by '+'
procedure UrlEncodeName(W: TTextWriter; const Text: RawUtf8); overload;
Append
a string as URI network name encoding, i.e. ' ' as %20
- only parameters - i.e. after '?' - should replace spaces by '+'
function UrlEncodeName(const Text: RawUtf8): RawUtf8; overload;
Encode a string as URI network name encoding, i.e. ' ' as %20
- only parameters - i.e. after '?' - should replace spaces by '+'
function UrlEncodeName(Text: PUtf8Char): RawUtf8; overload;
Encode a string as URI network name encoding, i.e. ' ' as %20
- only parameters - i.e. after '?' - should replace spaces by '+'
AlgoRle: TAlgoCompress;
Run-Length-Encoding (RLE) compression as a TAlgoCompress
class
- if RLE has no effect during compression, will fallback to plain store
AlgoRleLZ: TAlgoCompress;
SynLZ compression with RLE preprocess as a TAlgoCompress
class
- SynLZ is not efficient when its input has a lot of identical characters (e.g. a database content, or a raw binary buffer) - try with this class which is slower than AlgoSynLZ
but may have better ratio on such content
AlgoSynLZ: TAlgoCompress;
Our fast SynLZ compression as a TAlgoCompress
class
- please use this global variable methods instead of the deprecated SynLZCompress
/SynLZDecompress
wrapper functions
Base64DecodeMain: function(sp, rp: PAnsiChar; len: PtrInt): boolean;
Raw function for efficient Base64 to binary decoding of the main block
- don't use this function, but rather the Base64ToBin
() overloaded functions
- on FPC x86_64, detect and use AVX2 asm for very high throughput (9GB/s)
Base64EncodeMain: function(rp, sp: PAnsiChar; len: cardinal): integer;
Raw function for efficient binary to Base64 encoding of the main block
- don't use this function, but rather the BinToBase64
() overloaded functions
- on FPC x86_64, detect and use AVX2 asm for very high throughput (11GB/s)
EMOJI_AFTERDOTS: array['('..'|'] of TEmoji;
To recognize simple :) :( :| :/ :D :o :p :s characters as smilleys
EMOJI_RTTI: PShortString;
Low-level access to TEmoji
RTTI
- used when inlining EmojiFromText
()
EMOJI_TAG: array[TEmoji] of RawUtf8;
Github/Markdown compatible tag of Emojis, including trailing and ending :
- e.g. ':grinning:' or ':person_with_pouting_face:'
EMOJI_TEXT: array[TEmoji] of RawUtf8;
Github/Markdown compatible text
of Emojis
- e.g. 'grinning' or 'person_with_pouting_face'
EMOJI_UTF8: array[TEmoji] of RawUtf8;
The Unicode character matching a given Emoji, after UTF-8 encoding
LogCompressAlgo: TAlgoCompress;
Define how files are compressed by TSynLog.PerformRotation
- as used within mormot.core.log.pas unit, and defined in this unit to be available wihout any dependency to it (e.g. in compression units)
- assigned to AlgoSynLZ
by default for .synlz which is the fastest for logs
- you may set AlgoGZFast
from mormot.core.zip.pas to generate .gz standard files during TSynLog
file rotation (with libdeflate if available)
- you may set AlgoLizardFast
or AlgoLizardHuffman
as non-standard alternatives (default AlgoLizard
is much slower and less efficient on logs)
- if you set nil, no compression will take place during rotation
- note that compression itself is run in the logging background thread
LogCompressAlgoArchive: function(aAlgo: TAlgoCompress; aMagic: cardinal; const aOldLogFileName, aDestinationPath: TFileName): boolean;
Internal wrapper function used by TSynLogArchiveEvent
handlers to compress and delete older .log files using our proprietary FileCompress format for a given algorithm
- as used within mormot.core.log.pas unit, and defined in this unit to be available wihout any dependency to it (e.g. in compression units)
- called by EventArchiveLizard
/EventArchiveSynLZ
to implement .synlz/.synliz archival