doc/Encoding.3 - main - niceSOFT/tcl

doc/Encoding.3
665 строк33 KB
jan.nijtmans
Fix html generation: Encoding: process-text: impotent font change
24 дек 2025, 20:07
24 дек 2025, 20:075b3eb9f
'\"
'\" Copyright (c) 1997-1998 Sun Microsystems, Inc.
'\"
'\" See the file "license.terms" for information on usage and redistribution
'\" of this file, and for a DISCLAIMER OF ALL WARRANTIES.
'\"
.TH Tcl_GetEncoding 3 "8.1" Tcl "Tcl Library Procedures"
.so man.macros
.BS
.SH NAME
Tcl_GetEncoding, Tcl_FreeEncoding, Tcl_GetEncodingFromObj, Tcl_ExternalToUtfDString, Tcl_ExternalToUtfDStringEx, Tcl_ExternalToUtf,Tcl_ExternalToUtfEx, Tcl_UtfToExternalDString, Tcl_UtfToExternalDStringEx, Tcl_UtfToExternal, Tcl_UtfToExternalEx, Tcl_GetEncodingName, Tcl_SetSystemEncoding, Tcl_GetEncodingNameFromEnvironment, Tcl_GetEncodingNameForUser, Tcl_GetEncodingNames, Tcl_CreateEncoding, Tcl_GetEncodingSearchPath, Tcl_SetEncodingSearchPath \- procedures for creating and using encodings
.SH SYNOPSIS
.nf
\fB#include <tcl.h>\fR
.sp
Tcl_Encoding
\fBTcl_GetEncoding\fR(\fIinterp, name\fR)
.sp
\fBTcl_FreeEncoding\fR(\fIencoding\fR)
.sp
int
\fBTcl_GetEncodingFromObj\fR(\fIinterp, objPtr, encodingPtr\fR)
.sp
char *
\fBTcl_ExternalToUtfDString\fR(\fIencoding, src, srcLen, dstPtr\fR)
.sp
int
\fBTcl_ExternalToUtfDStringEx\fR(\fIinterp, encoding, src, srcLen, flags, dstPtr, errorIdxPtr\fR)
.sp
char *
\fBTcl_UtfToExternalDString\fR(\fIencoding, src, srcLen, dstPtr\fR)
.sp
int
\fBTcl_UtfToExternalDStringEx\fR(\fIinterp, encoding, src, srcLen, flags, dstPtr, errorIdxPtr\fR)
.sp
int
\fBTcl_ExternalToUtfEx\fR(\fIinterp, encoding, src, srcLen, flags, statePtr, dst, dstLen, srcReadPtr, dstWrotePtr, dstCharsPtr\fR)
.sp
int
\fBTcl_ExternalToUtf\fR(\fIinterp, encoding, src, srcLen, flags, statePtr, dst, dstLen, srcReadIntPtr, dstWroteIntPtr, dstCharsIntPtr\fR)
.sp
int
\fBTcl_UtfToExternalEx\fR(\fIinterp, encoding, src, srcLen, flags, statePtr, dst, dstLen, srcReadPtr, dstWrotePtr, dstCharsPtr\fR)
.sp
int
\fBTcl_UtfToExternal\fR(\fIinterp, encoding, src, srcLen, flags, statePtr, dst, dstLen, srcReadIntPtr, dstWroteIntPtr, dstCharsIntPtr\fR)
.sp
const char *
\fBTcl_GetEncodingName\fR(\fIencoding\fR)
.sp
Tcl_Size
\fBTcl_GetEncodingNulLength\fR(\fIencoding\fR)
.sp
int
\fBTcl_SetSystemEncoding\fR(\fIinterp, name\fR)
.sp
const char *
\fBTcl_GetEncodingNameFromEnvironment\fR(\fIbufPtr\fR)
.sp
const char *
\fBTcl_GetEncodingNameForUser\fR(\fIbufPtr\fR)
.sp
\fBTcl_GetEncodingNames\fR(\fIinterp\fR)
.sp
Tcl_Encoding
\fBTcl_CreateEncoding\fR(\fItypePtr\fR)
.sp
Tcl_Obj *
\fBTcl_GetEncodingSearchPath\fR()
.sp
int
\fBTcl_SetEncodingSearchPath\fR(\fIsearchPath\fR)
.fi
.SH ARGUMENTS
.AS "const Tcl_EncodingType" *dstWroteIntPtr in/out
.AP Tcl_Interp *interp in
Interpreter to use for error reporting, or NULL if no error reporting is
desired.
.AP "const char" *name in
Name of encoding to load.
.AP Tcl_Encoding encoding in
The encoding to query, free, or use for converting text.  If \fIencoding\fR is
NULL, the current system encoding is used.
.AP Tcl_Obj *objPtr in
Name of encoding to get token for.
.AP Tcl_Encoding *encodingPtr out
Points to storage where encoding token is to be written.
.AP "const char" *src in
For the \fBTcl_ExternalToUtf\fR functions, an array of bytes in the
specified encoding that are to be converted to TUTF-8.  For the
\fBTcl_UtfToExternal\fR function, a TUTF-8 byte sequence
to be converted to the specified encoding.
.AP "const TCHAR" *tsrc in
An array of Windows TCHAR characters to convert to TUTF-8.
.AP Tcl_Size srcLen in
Length of \fIsrc\fR or \fItsrc\fR in bytes.  If the length is negative, the
encoding-specific length of the string is used.
.AP Tcl_DString *dstPtr out
Pointer to an uninitialized or free \fBTcl_DString\fR in which the converted
result will be stored.
.AP int flags in
This is a bit mask passed in to control the operation of the encoding functions.
Any bits not defined in the function descriptions below should be set to 0 as they
are used internally by Tcl.
.AP Tcl_EncodingState *statePtr in/out
Used when converting a (generally long or indefinite length) byte stream
in a piece-by-piece fashion.
.AP char *dst out
Buffer in which the converted result will be stored.  No more than
\fIdstLen\fR bytes will be stored in \fIdst\fR.
.AP Tcl_Size dstLen in
The maximum length of the output buffer \fIdst\fR in bytes.
.AP Tcl_Size *srcReadPtr out
Filled with the number of bytes from \fIsrc\fR that were converted. May be NULL.
.AP int *srcReadIntPtr out
Filled with the number of bytes from \fIsrc\fR that were converted. May be NULL.
.AP Tcl_Size *dstWrotePtr out
Filled with the number of bytes that were stored in the output
buffer as a result of the conversion. May be NULL.
.AP int *dstWroteIntPtr out
Filled with the number of bytes that were stored in the output
buffer as a result of the conversion.  May be NULL.
.AP Tcl_Size *dstCharsPtr out
Filled with the number of characters that correspond to the number of bytes
stored in the output buffer. May be NULL.
.AP int *dstCharsIntPtr out
Filled with the number of characters that correspond to the number of bytes
stored in the output buffer. May be NULL.
.AP Tcl_Size *errorIdxPtr out
Filled with the index of the byte or character that caused the encoding transform
to fail. May be NULL.
.AP Tcl_DString *bufPtr out
Storage for the prescribed system encoding name.
.AP "const Tcl_EncodingType" *typePtr in
Structure that defines a new type of encoding.
.AP Tcl_Obj *searchPath in
List of filesystem directories in which to search for encoding data files.
.AP "const char" *path in
A path to the location of the encoding file.
.BE
.SH INTRODUCTION
.PP
N.B. Refer to the \fBTcl_UniChar\fR documentation page for a description of the
\fITUTF-8\fR encoding and related terms referenced here.
.PP
These routines convert between TUTF-8
and character representations using encodings such as
standard UTF-8, UTF-16, ASCII, or Shift-JIS that might be expected by system
interfaces or other software components. For instance, on a Japanese Unix
workstation, a user might obtain a filename represented in the EUC-JP file
encoding and then translate the characters to the jisx0208 font encoding in
order to display the filename in a Tk widget. The purpose of the encoding
package is to help bridge the translation gap. TUTF-8 provides an intermediate
staging ground for all the various encodings. In the example above, text would
be translated into TUTF-8 from whatever file encoding the operating system is
using. Then it would be translated from TUTF-8 into whatever font encoding the
display routines require.
.PP
Some basic encodings are compiled into Tcl. Others can be defined by the user or
dynamically loaded from encoding files in a platform-independent manner.
.SH "MANAGING ENCODINGS"
.PP
\fBTcl_GetEncoding\fR finds an encoding given its \fIname\fR. The name may refer
to a built-in Tcl encoding, a user-defined encoding registered by calling
\fBTcl_CreateEncoding\fR, or a dynamically-loadable encoding file. The return
value is a token that represents the encoding and can be used in subsequent
calls to functions that expect an argument of type \fBTcl_Encoding\fR. If the
name did not refer to any known or loadable encoding, NULL is returned and an
error message is stored in \fIinterp\fR.
.PP
The encoding package maintains a database of all encodings currently in use.
The first time \fIname\fR is seen, \fBTcl_GetEncoding\fR returns an
encoding with a reference count of 1.  If the same \fIname\fR is requested
further times, then the reference count for that encoding is incremented
without the overhead of allocating a new encoding and all its associated
data structures.
.PP
When an \fIencoding\fR is no longer needed, \fBTcl_FreeEncoding\fR
should be called to release it.  When an \fIencoding\fR is no longer in use
anywhere (i.e., it has been freed as many times as it has been gotten)
\fBTcl_FreeEncoding\fR will release all storage the encoding was using
and delete it from the database.
.PP
\fBTcl_GetEncodingFromObj\fR treats the string representation of
\fIobjPtr\fR as an encoding name, and finds an encoding with that
name, just as \fBTcl_GetEncoding\fR does. When an encoding is found,
it is cached within the \fBobjPtr\fR value for future reference, the
\fBTcl_Encoding\fR token is written to the storage pointed to by
\fIencodingPtr\fR, and the value \fBTCL_OK\fR is returned. If no such
encoding is found, the value \fBTCL_ERROR\fR is returned, and no
writing to \fB*\fR\fIencodingPtr\fR takes place. Just as with
\fBTcl_GetEncoding\fR, the caller should call \fBTcl_FreeEncoding\fR
on the resulting encoding token when that token will no longer be
used.
.PP
\fBTcl_GetEncodingName\fR is roughly the inverse of \fBTcl_GetEncoding\fR.
Given an \fIencoding\fR, the return value is the \fIname\fR argument that
was used to create the encoding.  The string returned by
\fBTcl_GetEncodingName\fR is only guaranteed to persist until the
\fIencoding\fR is deleted.  The caller must not modify this string.
.PP
\fBTcl_GetEncodingNulLength\fR returns the length of the terminating
nul byte sequence for strings in the specified encoding.
.PP
\fBTcl_SetSystemEncoding\fR sets the default encoding that should be used
whenever the user passes a NULL value for the \fIencoding\fR argument to
any of the other encoding functions.  If \fIname\fR is NULL, the system
encoding is reset to the default system encoding, \fBbinary\fR.  If the
name did not refer to any known or loadable encoding, \fBTCL_ERROR\fR is
returned and an error message is left in \fIinterp\fR.  Otherwise, this
procedure increments the reference count of the new system encoding,
decrements the reference count of the old system encoding, and returns
\fBTCL_OK\fR.
.PP
\fBTcl_GetEncodingNameFromEnvironment\fR retrieves the encoding name to
use as the system encoding. On non-Windows platforms, this is derived
from the \fBnl_langinfo\fR system call if available, and environment
variables \fBLC_ALL\fR, \fBLC_CTYPE\fR or \fBLANG\fR otherwise. On
Windows versions Windows 10 Build 18362 and later the returned value is
always \fButf-8\fR. On earlier Windows versions, it is derived from the
user settings in the Windows registry. \fBTcl_GetEncodingNameForUser\fR
retrieves the encoding name based on the user settings for the current
user and is derived in the same manner as
\fBTcl_GetEncodingNameFromEnvironment\fR on non-Windows platforms. On
Windows, unlike \fBTcl_GetEncodingNameFromEnvironment\fR, it returns the
encoding name as per the Windows registry settings irrespective of the
Windows version. Both functions accept \fIbufPtr\fR, a pointer to an
uninitialized or freed \fBTcl_DString\fR and write the encoding name to
it. They return \fBTcl_DStringValue(bufPtr)\fR which points to the stored
name.
.PP
\fBTcl_GetEncodingNames\fR sets the \fIinterp\fR result to a list
consisting of the names of all the encodings that are currently defined
or can be dynamically loaded, searching the encoding path specified by
\fBTcl_SetEncodingSearchPath\fR.  This procedure does not ensure that the
dynamically-loadable encoding files contain valid data, but merely that they
exist.
.SH "CONVERTING BETWEEN TUTF-8 AND OTHER ENCODINGS"
.PP
There are two sets of functions to convert data between TUTF-8 and any external
encoding known to Tcl. They differ primarily in the form in which converted data
is returned to the caller. The \fBTcl_ExternalToUtfEx\fR and
\fBTcl_UtfToExternalEx\fR functions return the data in buffers supplied by the
caller. The \fBTcl_ExternalToUtfDStringEx\fR and
\fBTcl_UtfToExternalDStringEx\fR return the data in a \fBTcl_DString\fR
structure. For backwards compatibility, Tcl also provides "non-Ex" variants
of these such as \fBTcl_ExternalToUtf\fR.
.SS "CONVERSION USING OUTPUT BUFFERS"
.PP
The \fBTcl_ExternalToUtfEx\fR function converts bytes in the encoding
\fIencoding\fR at address \fIsrc\fR into TUTF-8 encoding, storing them in the
buffer at address \fIdst\fR. Conversely, \fBTcl_UtfToExternalEx\fR converts
bytes encoded in TUTF-8 at address \fIsrc\fR into the encoding given by
\fIencoding\fR, storing them in the output buffer at address \fIdst\fR. In both
cases, \fIsrcLen\fR specifies the number of bytes to be converted and
\fIdstLen\fR specifies the size of the output buffer.
.PP
The \fIflags\fR parameter is a bitmask that controls operation of the conversion
and should be a bitwise OR of zero or more of the following values:
.RS
.IP \fBTCL_ENCODING_PROFILE_xxx\fR 29
At most one of the profile selection flags listed in the \fBPROFILES\fR section of
this manpage.
.IP \fBTCL_ENCODING_NO_TERMINATE\fR 29
Disables null termination of the output. By default, the output buffer \fIdst\fR
is terminated with an encoding-appropriate null.
.IP \fBTCL_ENCODING_START\fR and \fBTCL_ENCODING_END\fR 29
Indicate whether the source bytes correspond to the first or last blocks,
respectively, in a source stream. \fBTCL_ENCODING_START\fR will cause the
conversion routine to reset to an initial state ready to process the first byte
of an encoded stream. \fBTCL_ENCODING_END\fR indicates the source buffer is the
last block in an input stream allowing any required finalization to be
performed. Any incomplete trailing characters will then be treated as per the
encoding profile in effect. Both flags may be specified in the same call when
all data to be converted is passed in a single block. Both flags are also
presumed to be implicitly set if the \fIstatePtr\fR parameter is passed as NULL.
.IP \fBTCL_ENCODING_CHAR_LIMIT\fR 29
Specifies that the functions should not convert more characters than the number
passed through the \fIdstCharsPtr\fR argument, if not NULL. This flag is only
supported by the \fBTcl_ExternalToUtfEx\fR function and should not be passed to
other functions.
.RE
.LP
The \fIstatePtr\fR parameter is an opaque pointer to a location used by the
encoding functions to store intermediate state when the data to be converted is
passed in multiple chunks. The same location should be passed in \fIstatePtr\fR
for all related calls with the first chunk passed with the
\fBTCL_ENCODING_START\fR flag set and the last with \fBTCL_ENCODING_END\fR set.
If \fIstatePtr\fR is passed as NULL, all data is presumed to all be contained in
that single call and the functions behave as if \fBTCL_ENCODING_START\fR and
\fBTCL_ENCODING_END\fR were both set in the \fIflags\fR parameter.
.PP
The \fIsrcReadPtr\fR, \fIdstWrotePtr\fR and \fIdstCharsPtr\fR point to locations
to hold the number of bytes in the source that were processed successfully, the
number of bytes written to the output buffer (excluding terminating nulls), and
the number of characters written to the output buffer (again excluding
terminating nulls) respectively. All three are optional and may be passed as
NULL. Further, in the case of the \fBTcl_ExternalToUtfEx\fR function, the
\fIdstCharsPtr\fR may be used with the \fBTCL_ENCODING_CHAR_LIMIT\fR flags to
limit the number of characters processed as described earlier.
.PP
With the exceptions noted below, the counts returned in \fIsrcReadPtr\fR,
\fIdstWrotePtr\fR and \fIdstCharsPtr\fR are valid for all return codes listed
below other than \fBTCL_ERROR\fR.
.RS
.IP \fBTCL_OK\fR 29
The function completed without any exceptional conditions. Note this does not
mean all passed input in \fIsrc\fR was processed or verified. In particular, in
the case of the caller passing \fBTCL_ENCODING_CHAR_LIMIT\fR to limit the number
of characters converted, only the corresponding number of bytes in the source
input would have been processed as indicated by the value in \fIsrcReadPtr\fR.
.IP \fBTCL_CONVERT_NOSPACE\fR 29
The output buffer had insufficient space. The output buffer will contain as much
converted data as it could fit and will be null terminated as appropriate unless
the buffer was too small to even contain a null terminator by itself.
\fIsrcReadPtr\fR will hold number of processed source bytes and caller should
call again to process the remaining bytes. *Note: As a quirk of implementation,
in some cases the destination buffer needs to be \fBTCL_UTF_MAX\fR bytes greater
than the actual size needed. This is an existing quirk present in both 8.6 and
9.0 that is not addressed in this TIP.*
.IP \fBTCL_CONVERT_MULTIBYTE\fR 29
The trailing bytes in the source input formed an incomplete encoding sequence.
Caller should call the function again with additional source bytes appended to
the tail at offset \fI*srcReadPtr\fR of the original source bytes and with the
same \fIstatePtr\fR. Note that if \fIflags\fR had the \fBTCL_ENCODING_END\fR
flag set, indicating no more data is forthcoming, the functions will return
\fBTCL_CONVERT_SYNTAX\fR instead of \fBTCL_CONVERT_MULTIBYTE\fR.
.IP \fBTCL_CONVERT_SYNTAX\fR 29
An invalid byte sequence was detected in the source input. What constitutes an
"invalid" sequence is subject to the encoding profile as specified by the
\fIflags\fR parameter. The \fI*srcReadPtr\fR count will contain the number of
bytes successfully processed and is therefore also the offset of the start of
the invalid sequence. The output buffer will contain the converted data up to
that point.
.IP \fBTCL_CONVERT_UNKNOWN\fR 29
The input byte sequence represented a character that cannot be encoded in the
output encoding for the encoding profile in effect. Treatment is similar to that
of \fBTCL_CONVERT_SYNTAX\fR.
.IP \fBTCL_ERROR\fR 29
An error message is stored in \fIinterp\fR if it is not NULL. The output
locations at \fIsrcReadPtr\fR, \fIdstWrotePtr\fR and \fIdstCharsPtr\fR may have
been modified but should not be considered valid.
.RE
.LP
The functions \fBTcl_ExternalToUtf\fR and \fBTcl_UtfToExternal\fR are variants
of \fBTcl_ExternalToUtfEx\fR and \fBTcl_UtfToExternalEx\fR. They differ in that
their output counts have a limit of INT_MAX and therefore cannot handle the full
string lengths supported by Tcl on 64-bit platforms. They will return
\fBTCL_ERROR\fR in such cases.
.SS "CONVERSION USING TCL_DSTRING"
.PP
\fBTcl_ExternalToUtfDStringEx\fR and \fBTcl_UtfToExternalDStringEx\fR convert
\fIsrcLen\fR bytes at address \fIsrc\fR from the specified \fIencoding\fR into
TUTF-8 and the reverse respectively. The converted bytes are stored in
\fIdstPtr\fR, which is then null-terminated. The caller should eventually call
\fBTcl_DStringFree\fR to free any information stored in \fIdstPtr\fR irrespective
of the return value from the function.
.PP
The \fIflags\fR argument to the functions should be 0 or one of the profile
selection flags described above to select the profile to use for conversion.
The other flags should be cleared. The functions assume the entire source
string to be converted is passed into the function.
.PP
On success, the function returns \fBTCL_OK\fR with the
converted string stored in \fB*dstPtr\fR. For errors \fIother than conversion
errors\fR, such as invalid flags, the function returns \fBTCL_ERROR\fR with an
error message in \fBinterp\fR if it is not NULL.
For conversion errors, \fBTcl_ExternalToUtfDStringEx\fR returns one
of the \fBTCL_CONVERT_*\fR errors listed above.
When one of these conversion errors is returned, an error message is stored
in \fBinterp\fR only if \fBerrorIdxPtr\fR is NULL. Otherwise, no error message
is stored as the function expects the caller is only interested the decoded data
up to that point and not treating this as an immediate error condition.
The index of the error location is stored in \fB*errorIdxPtr\fR.
.PP
\fBTcl_ExternalToUtfDString\fR and \fBTcl_UtfToExternalDString\fR are older,
less flexible variants of the above functions that do not support profiles. The
return value from the functions is a pointer to the value stored in the
\fBTcl_DString\fR. When converting, if any of the characters in the source
buffer cannot be represented in the target encoding, a default fallback
character will be used. The return value is a pointer to the value stored in the
DString. \fBTcl_UtfToExternalDString\fR converts a source buffer \fIsrc\fR from
TUTF-8 into the specified \fIencoding\fR. The converted bytes are stored in
\fIdstPtr\fR, which is then terminated with the appropriate encoding-specific
null. The caller should eventually call \fBTcl_DStringFree\fR to free any
information stored in \fIdstPtr\fR. When converting, if any of the characters in
the source buffer cannot be represented in the target encoding, an
encoding-specific default fallback character will be used.
.SH CREATING NEW ENCODINGS
.PP
\fBTcl_CreateEncoding\fR defines a new encoding and registers the C
procedures that are called back to convert between the encoding and
TUTF-8.  Encodings created by \fBTcl_CreateEncoding\fR are thereafter
visible in the database used by \fBTcl_GetEncoding\fR.  Just as with the
\fBTcl_GetEncoding\fR procedure, the return value is a token that
represents the encoding and can be used in subsequent calls to other
encoding functions.  \fBTcl_CreateEncoding\fR returns an encoding with a
reference count of 1. If an encoding with the specified \fIname\fR
already exists, then its entry in the database is replaced with the new
encoding; the token for the old encoding will remain valid and continue
to behave as before, but users of the new token will now call the new
encoding procedures.
.PP
The \fItypePtr\fR argument to \fBTcl_CreateEncoding\fR contains information
about the name of the encoding and the procedures that will be called to
convert between this encoding and TUTF-8.  It is defined as follows:
.PP
.CS
typedef struct {
    const char *\fIencodingName\fR;
    Tcl_EncodingConvertProc *\fItoUtfProc\fR;
    Tcl_EncodingConvertProc *\fIfromUtfProc\fR;
    Tcl_EncodingFreeProc *\fIfreeProc\fR;
    void *\fIclientData\fR;
    Tcl_Size \fInullSize\fR;
} \fBTcl_EncodingType\fR;
.CE
.PP
The \fIencodingName\fR provides a string name for the encoding, by
which it can be referred in other procedures such as
\fBTcl_GetEncoding\fR.  The \fItoUtfProc\fR refers to a callback
procedure to invoke to convert text from this encoding into TUTF-8.
The \fIfromUtfProc\fR refers to a callback procedure to invoke to
convert text from TUTF-8 into this encoding.  The \fIfreeProc\fR refers
to a callback procedure to invoke when this encoding is deleted.  The
\fIfreeProc\fR field may be NULL.  The \fIclientData\fR contains an
arbitrary one-word value passed to \fItoUtfProc\fR, \fIfromUtfProc\fR,
and \fIfreeProc\fR whenever they are called.  Typically, this is a
pointer to a data structure containing encoding-specific information
that can be used by the callback procedures.  For instance, two very
similar encodings such as \fBascii\fR and \fBmacRoman\fR may use the
same callback procedure, but use different values of \fIclientData\fR
to control its behavior.  The \fInullSize\fR specifies the number of
zero bytes that signify end-of-string in this encoding.  It must be
\fB1\fR (for single-byte or multi-byte encodings like ASCII or
Shift-JIS) or \fB2\fR (for double-byte encodings like Unicode).
Constant-sized encodings with 3 or more bytes per character (such as
CNS11643) are not accepted.
.PP
The callback procedures \fItoUtfProc\fR and \fIfromUtfProc\fR should match the
type \fBTcl_EncodingConvertProc\fR:
.PP
.CS
typedef int \fBTcl_EncodingConvertProc\fR(
        void *\fIclientData\fR,
        const char *\fIsrc\fR,
        int \fIsrcLen\fR,
        int \fIflags\fR,
        Tcl_EncodingState *\fIstatePtr\fR,
        char *\fIdst\fR,
        int \fIdstLen\fR,
        int *\fIsrcReadIntPtr\fR,
        int *\fIdstWroteIntPtr\fR,
        int *\fIdstCharsIntPtr\fR);
.CE
.PP
The \fItoUtfProc\fR and \fIfromUtfProc\fR procedures are called by the
\fBTcl_ExternalToUtf\fR or \fBTcl_UtfToExternal\fR family of functions to
perform the actual conversion.  The \fIclientData\fR parameter to these
procedures is the same as the \fIclientData\fR field specified to
\fBTcl_CreateEncoding\fR when the encoding was created.  The remaining
arguments to the callback procedures are the same as the arguments,
documented at the top, to \fBTcl_ExternalToUtf\fR or
\fBTcl_UtfToExternal\fR, with the following exceptions.  If the
\fIsrcLen\fR argument to one of those high-level functions is negative,
the value passed to the callback procedure will be the appropriate
encoding-specific string length of \fIsrc\fR.  The \fIsrcReadIntPtr\fR,
\fIdstWroteIntPtr\fR, and \fIdstCharsIntPtr\fR arguments will always be non-NULL,
even if the corresponding argument to one of the high-level
functions is NULL.
.PP
The callback procedure \fIfreeProc\fR, if non-NULL, should match the type
\fBTcl_EncodingFreeProc\fR:
.PP
.CS
typedef void \fBTcl_EncodingFreeProc\fR(
        void *\fIclientData\fR);
.CE
.PP
This \fIfreeProc\fR function is called when the encoding is deleted.  The
\fIclientData\fR parameter is the same as the \fIclientData\fR field
specified to \fBTcl_CreateEncoding\fR when the encoding was created.
.PP
\fBTcl_GetEncodingSearchPath\fR and \fBTcl_SetEncodingSearchPath\fR
are called to access and set the list of filesystem directories searched
for encoding data files.
.PP
The value returned by \fBTcl_GetEncodingSearchPath\fR
is the value stored by the last successful call to
\fBTcl_SetEncodingSearchPath\fR.  If no calls to
\fBTcl_SetEncodingSearchPath\fR have occurred, Tcl will compute an initial
value based on the environment.  There is one encoding search path for the
entire process, shared by all threads in the process.
.PP
\fBTcl_SetEncodingSearchPath\fR stores \fIsearchPath\fR and returns
\fBTCL_OK\fR, unless \fIsearchPath\fR is not a valid Tcl list, which
causes \fBTCL_ERROR\fR to be returned.  The elements of \fIsearchPath\fR
are not verified as existing readable filesystem directories.  When
searching for encoding data files takes place, and non-existent or
non-readable filesystem directories on the \fIsearchPath\fR are silently
ignored.
.SH "ENCODING FILES"
Space would prohibit precompiling into Tcl every possible encoding
algorithm, so many encodings are stored on disk as dynamically-loadable
encoding files.  This behavior also allows the user to create additional
encoding files that can be loaded using the same mechanism.  These
encoding files contain information about the tables and/or escape
sequences used to map between an external encoding and Unicode.  The
external encoding may consist of single-byte, multi-byte, or double-byte
characters.
.PP
Each dynamically-loadable encoding is represented as a text file.  The
initial line of the file, beginning with a
.QW #
symbol, is a comment
that provides a human-readable description of the file.  The next line
identifies the type of encoding file.  It can be one of the following
letters:
.IP "[1] \fBS\fR"
A single-byte encoding, where one character is always one byte long in the
encoding.  An example is \fBiso8859-1\fR, used by many European languages.
.IP "[2] \fBD\fR"
A double-byte encoding, where one character is always two bytes long in the
encoding.  An example is \fBbig5\fR, used for Chinese text.
.IP "[3] \fBM\fR"
A multi-byte encoding, where one character may be either one or two bytes long.
Certain bytes are lead bytes, indicating that another byte must follow
and that together the two bytes represent one character.  Other bytes are not
lead bytes and represent themselves.  An example is \fBshiftjis\fR, used by
many Japanese computers.
.IP "[4] \fBE\fR"
An escape-sequence encoding, specifying that certain sequences of bytes
do not represent characters, but commands that describe how following bytes
should be interpreted.
.PP
The rest of the lines in the file depend on the type.
.PP
Cases [1], [2], and [3] are collectively referred to as table-based encoding
files.  The lines in a table-based encoding file are in the same
format as this example taken from the \fBshiftjis\fR encoding (this is not
the complete file):
.PP
.CS
# Encoding file: shiftjis, multi-byte
M
003F 0 40
00
0000000100020003000400050006000700080009000A000B000C000D000E000F
0010001100120013001400150016001700180019001A001B001C001D001E001F
0020002100220023002400250026002700280029002A002B002C002D002E002F
0030003100320033003400350036003700380039003A003B003C003D003E003F
0040004100420043004400450046004700480049004A004B004C004D004E004F
0050005100520053005400550056005700580059005A005B005C005D005E005F
0060006100620063006400650066006700680069006A006B006C006D006E006F
0070007100720073007400750076007700780079007A007B007C007D203E007F
0080000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000
0000FF61FF62FF63FF64FF65FF66FF67FF68FF69FF6AFF6BFF6CFF6DFF6EFF6F
FF70FF71FF72FF73FF74FF75FF76FF77FF78FF79FF7AFF7BFF7CFF7DFF7EFF7F
FF80FF81FF82FF83FF84FF85FF86FF87FF88FF89FF8AFF8BFF8CFF8DFF8EFF8F
FF90FF91FF92FF93FF94FF95FF96FF97FF98FF99FF9AFF9BFF9CFF9DFF9EFF9F
0000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000
81
0000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000
300030013002FF0CFF0E30FBFF1AFF1BFF1FFF01309B309C00B4FF4000A8FF3E
FFE3FF3F30FD30FE309D309E30034EDD30053006300730FC20152010FF0F005C
301C2016FF5C2026202520182019201C201DFF08FF0930143015FF3BFF3DFF5B
FF5D30083009300A300B300C300D300E300F30103011FF0B221200B100D70000
00F7FF1D2260FF1CFF1E22662267221E22342642264000B0203220332103FFE5
FF0400A200A3FF05FF03FF06FF0AFF2000A72606260525CB25CF25CE25C725C6
25A125A025B325B225BD25BC203B301221922190219121933013000000000000
000000000000000000000000000000002208220B2286228722822283222A2229
000000000000000000000000000000002227222800AC21D221D4220022030000
0000000000000000000000000000000000000000222022A52312220222072261
2252226A226B221A223D221D2235222B222C0000000000000000000000000000
212B2030266F266D266A2020202100B6000000000000000025EF000000000000
.CE
.PP
The third line of the file is three numbers.  The first number is the
fallback character (in base 16) to use when converting from TUTF-8 to this
encoding.  The second number is a \fB1\fR if this file represents the
encoding for a symbol font, or \fB0\fR otherwise.  The last number (in base
10) is how many pages of data follow.
.PP
Subsequent lines in the example above are pages that describe how to map
from the encoding into 2-byte Unicode.  The first line in a page identifies
the page number.  Following it are 256 double-byte numbers, arranged as 16
rows of 16 numbers.  Given a character in the encoding, the high byte of
that character is used to select which page, and the low byte of that
character is used as an index to select one of the double-byte numbers in
that page \- the value obtained being the corresponding Unicode character.
By examination of the example above, one can see that the characters 0x7E
and 0x8163 in \fBshiftjis\fR map to 203E and 2026 in Unicode, respectively.
.PP
Following the first page will be all the other pages, each in the same
format as the first: one number identifying the page followed by 256
double-byte Unicode characters.  If a character in the encoding maps to the
Unicode character 0000, it means that the character does not actually exist.
If all characters on a page would map to 0000, that page can be omitted.
.PP
Case [4] is the escape-sequence encoding file.  The lines in an this type of
file are in the same format as this example taken from the \fBiso2022-jp\fR
encoding:
.PP
.CS
.ta 1.5i
# Encoding file: iso2022-jp, escape-driven
E
init		{}
final		{}
iso8859-1	\ex1B(B
jis0201		\ex1B(J
jis0208		\ex1B$@
jis0208		\ex1B$B
jis0212		\ex1B$(D
gb2312		\ex1B$A
ksc5601		\ex1B$(C
.CE
.PP
In the file, the first column represents an option and the second column
is the associated value.  \fBinit\fR is a string to emit or expect before
the first character is converted, while \fBfinal\fR is a string to emit
or expect after the last character.  All other options are names of
table-based encodings; the associated value is the escape-sequence that
marks that encoding.  Tcl syntax is used for the values; in the above
example, for instance,
.QW \fB{}\fR
represents the empty string and
.QW \fB\ex1B\fR
represents character 27.
.PP
When \fBTcl_GetEncoding\fR encounters an encoding \fIname\fR that has not
been loaded, it attempts to load an encoding file called \fIname\fB.enc\fR
from the \fBencoding\fR subdirectory of each directory that Tcl searches
for its script library.  If the encoding file exists, but is
malformed, an error message will be left in \fIinterp\fR.
.SH "REFERENCE COUNT MANAGEMENT"
.PP
\fBTcl_GetEncodingFromObj\fR does not modify the reference count of its
\fIobjPtr\fR argument; it only reads. Note however that this function may set
the interpreter result; if that is the only place that is holding a reference
to the object, it will be deleted.
.PP
\fBTcl_GetEncodingSearchPath\fR returns an object with a reference count of at
least 1.
.SH "PROFILES"
Encoding profiles define the manner in which errors in the encoding transforms
are handled by the encoding functions. An application can specify the profile
to be used by OR-ing the \fBflags\fR parameter passed to the function
with at most one of \fBTCL_ENCODING_PROFILE_TCL8\fR,
\fBTCL_ENCODING_PROFILE_STRICT\fR or \fBTCL_ENCODING_PROFILE_REPLACE\fR.
These correspond to the \fBtcl8\fR, \fBstrict\fR and \fBreplace\fR profiles
respectively. If none are specified, a version-dependent default profile is used.
For Tcl 9.0, the default profile is \fBstrict\fR.
.PP
For details about profiles, see the \fBPROFILES\fR section in
the documentation of the \fBencoding\fR command.
.SH "SEE ALSO"
encoding(n)
.SH KEYWORDS
utf, encoding, convert