Provides encoding-agnostic Unicode string manipulation functions and an enumerator.

lazunicode.pas provides encoding-agnostic Unicode string manipulation functions and an enumerator. It works transparently with UTF-8 and UTF-16 encodings, and allows one codebase to work for:

1. Lazarus using its default UTF-8 encoding
2. Future FPC and Lazarus versions with Delphi compatible UTF-16 encoding
3. Delphi compatibility where String is defined as UnicodeString

Behavior of helper functions are altered using the {$ModeSwitch UnicodeStrings} directive; the correct routines for handling UTF-8 or UTF-16 are called based on the mode switch value.

This file is part of the LazUtils package.

Copies the specified number of codepoints starting at a character position.

Copies the number of codepoints in CharCount from s, starting at the character position in StartCharIndex. For platforms that require UTF-16, UTF16Copy is called. For other platforms, UTF8Copy is called.

UTF16Copy UTF8Copy
Values copied from the string.
UTF-encoded string values.
Initial character position.
Number of characters needed in the copy operation.
Gets the number of codepoints in the specified string.
Gets the number of codepoints in the specified string. For platforms that require UTF-16, UTF16Length is called to get the return value for the function. For other platforms, UTF8LengthFast is called to get the number of codepoints.
Number of codepoints in the string.
UTF-encoded values examined in the function.
Gets the position where the search value is found in a string.

Gets the position in SearchInText where SearchForText is found. StartPos indicates the initial character position (codepoint) in SearchInText used for the comparison. The default value is 1.

The return value contains the character position (codepoint) where the search value was found. The return value is 0 (zero) if SearchForText is not found in the string. For platforms that require UTF-16, UTF16Pos is called to get the return value. For other platforms, UTF8Pos is called to get the character position (codepoint).

Character position (codepoint) where the search value was found in the string.
Values to locate in the string.
String to search for the specified values.
Initial character position (codepoint) used in the comparison.
Gets the number of bytes needed for a CodePoint in the specified value.
Gets the number of bytes needed for the CodePoint specified in p. For platforms that require UTF-16, TCharacter.IsHighSurrogate is called to get the return value. For other platforms, UTF8CodepointSizeFast is called to get the number of bytes for the codepoint. The return value is 1 or 2 for UTF-16-enabled platforms, or in the range 1..4 for UTF-8-enabled platforms. The return value can be 0 (zero) if p contains an empty string ('') or a malformed codepoint.
Number of bytes required for a codepoint.
String with the codepoint to examine in the function.
Determines if the specified value is a combining codepoint.
Determines if the specified value is a combining codepoint. Please note, there are many more rules for combining codepoints.The diacritical marks handled in the function are only a subset of the possible Unicode values. For platforms that require UTF-16, UTF16IsCombining is called to get the return value for the specified codepoint. For other platforms, UTF8IsCombining is called to examine the codepoint.
True when the codepoint represents a Unicode combining character.
Codepoint to examine in the function.
Converts the specified value to the Windows system codepage.
Converts the specified value to the Windows system codepage. The Unicode encoding used in s depends on the modeswitch value. For platforms that require UTF-16, UTF16ToUTF8 and UTF8ToWinCP are called to get the return value for the function, except when String is defined as UnicodeString. No conversion is required in that situation. For other platforms, UTF8ToWinCP is called to get the return value.
Values after conversion to the Windows code page.
Unicode values to convert in the function.
Converts the specified string to Unicode.
Converts the specified value from the Windows system codepage to Unicode. The Unicode encoding used depends on the modeswitch value. For platforms that require UTF-16, WinCPToUTF8 and UTF8ToUTF16 are called to get the return value for the function. Except when String is defined as UnicodeString. No conversion is required in that situation. For other platforms, WinCPToUTF8 is called to get the return value.
Unicode values for the specified string.
String with Windows code page values.
Creates a string with the specified number of codepoints.
Creates a string with the specified number of codepoints. Like StringOfChar. For platforms that require UTF-16, the values in ACodePoint are concatenated together until the number of codepoints in N have been created. For other platforms, Utf8StringOfChar is called to get the return value for the function.
String with the specified number of codepoints.
Codepoint to use when creating the string.
Number of codepoints required in the string.
Base class for a Unicode character enumerator.
Base class for a Unicode character enumerator.
Copies byte values for the Current character (codepoint).
Copies byte values used in Current for the character (codepoint) when MoveNext is called to go to the next character. aCount contains the number of byte values needed for the Unicode codepoint. UpdateCurrent increments the internal pointer used to access values in the enumerator by the number of bytes in aCount.

Raises an assertion error if the number of bytes in aCount is 0 (zero). Raised with the message 'TUnicodeEnumeratorBase.UpdateCurrent: aCount=0'.

Raises an assertion error if the length of bytes copied to Current is different that the value in aCount. Raised with the message 'TUnicodeEnumeratorBase.UpdateCurrent: Length(fCurrent)<>aCount.')'.

Number of bytes needed for the codepoint.
Constructor for the class instance.
Create initializes internal member variable used to access byte values for Unicode codepoints. A is the string with codepoints traversed using the enumerator.
Unicode string for the enumerator.
Byte values for the current codepoint in the enumerator.
Current is a read-only String property which provides access to the byte values for the current codepoint in the enumerator. Current is updated in UpdateCurrent when the MoveNext method is called.
Number of bytes in the Current codepoint.
CurrentCodeUnitCount is a read-only Integer property which contains the number of bytes needed for the codepoint in Current. CurrentCodeUnitCount is updated in UpdateCurrent when MoveNext is called.
Base class for a Unicode codepoint enumerator.
Base class for a Unicode codepoint enumerator. TCodePointEnumerator allows traversal of Unicode codepoints. Uses UTF-8 or UTF-16 encodings depending on value in $ModeSwitch. Extends the ancestor class to provide navigation in the enumerator using the MoveNext method.
Provides navigation to the next codepoint in the enumerator.
Provides navigation to the next Unicode codepoint in the enumerator. The return value contains True when more characters (codepoints) are available to the enumerator. UpdateCurrent is called using the value from CodePointSize to store the value for the Current property.
True when more characters (codepoints) are available.
Implements an enumerator for Unicode codepoints.
Implements an enumerator for Unicode codepoints. TUnicodeCharacterEnumerator allows traversal of characters (codepoints) in a Unicode-encoded string. Values use either UTF-16 or UTF-8 encoding depending on the value for $ModeSwitch. An overridden MoveNext method is provided to handle combining diacritical marks in the Unicode codepoints.
Number of bytes used for the Current codepoint.
CurrentCodePointCount is a read-only Integer property that indicates the number of bytes used for the Current codepoint. CurrentCodePointCount is updated in the MoveNext method, and includes any combining diacritical marks found in the codepoints.
Adds support for combining diacritical marks when moving to the next codepoint.

MoveNext is an overridden method which adds support for combining diacritical marks when moving to the next codepoint for the enumerator. The return value is True when more characters (codepoints) are available to the enumerator. MoveNext updates the value in CurrentCodeUnitCount, and includes combining diacritical marks in the byte count. MoveNext calls UpdateCurrent to store the value for the Current property.

MoveNext does not call the inherited method.
True when more characters (codepoints) are available to the enumerator.
Enumerator which combines diacritical marks.

The enumerator operator enables For ... In loops. This enumerator combines diacritical marks in the String argument for the operator. It is used by default although there are more rules for combining codepoints. Diacritical marks cover rules for most western languages.