diff options
| author | Ulrich Drepper <drepper@redhat.com> | 2001-06-06 12:55:46 +0000 |
|---|---|---|
| committer | Ulrich Drepper <drepper@redhat.com> | 2001-06-06 12:55:46 +0000 |
| commit | 601d294296e1de9fc423700db00bccc04a2bf50d (patch) | |
| tree | 1d274eb4e512431b042e07bf8f8ff4a980407257 /iconvdata | |
| parent | 9a1f675436c5622b7846faa25274a2ef2d743285 (diff) | |
| download | glibc-601d294296e1de9fc423700db00bccc04a2bf50d.tar.gz glibc-601d294296e1de9fc423700db00bccc04a2bf50d.tar.xz glibc-601d294296e1de9fc423700db00bccc04a2bf50d.zip | |
Update.
2001-06-04 Bruno Haible <haible@clisp.cons.org> * iconv/loop.c (UNICODE_TAG_HANDLER): New macro. * iconv/gconv_simple.c (__gconv_transform_internal_ascii): Invoke UNICODE_TAG_HANDLER. (__gconv_transform_internal_ucs2): Likewise. (__gconv_transform_internal_ucs2reverse): Likewise. * iconvdata/8bit-gap.c (BODY for TO_LOOP): Invoke UNICODE_TAG_HANDLER. * iconvdata/8bit-generic.c (BODY for TO_LOOP): Likewise. * iconvdata/ansi_x3.110.c (BODY for TO_LOOP): Likewise. * iconvdata/big5.c (BODY for TO_LOOP): Likewise. * iconvdata/big5hkscs.c (BODY for TO_LOOP): Likewise. * iconvdata/cp1255.c (BODY for TO_LOOP): Likewise. * iconvdata/cp1258.c (BODY for TO_LOOP): Likewise. * iconvdata/euc-cn.c (BODY for TO_LOOP): Likewise. * iconvdata/euc-jp.c (BODY for TO_LOOP): Likewise. * iconvdata/euc-kr.c (BODY for TO_LOOP): Likewise. * iconvdata/euc-tw.c (BODY for TO_LOOP): Likewise. * iconvdata/gbk.c (BODY for TO_LOOP): Likewise. * iconvdata/ibm930.c (BODY for TO_LOOP): Likewise. * iconvdata/ibm932.c (BODY for TO_LOOP): Likewise. * iconvdata/ibm933.c (BODY for TO_LOOP): Likewise. * iconvdata/ibm935.c (BODY for TO_LOOP): Likewise. * iconvdata/ibm937.c (BODY for TO_LOOP): Likewise. * iconvdata/ibm939.c (BODY for TO_LOOP): Likewise. * iconvdata/ibm943.c (BODY for TO_LOOP): Likewise. * iconvdata/iso646.c (BODY for TO_LOOP): Likewise. * iconvdata/iso8859-1.c (BODY for TO_LOOP): Likewise. * iconvdata/iso_6937.c (BODY for TO_LOOP): Likewise. * iconvdata/iso_6937-2.c (BODY for TO_LOOP): Likewise. * iconvdata/iso-2022-cn.c (BODY for TO_LOOP): Likewise. * iconvdata/iso-2022-cn-ext.c (BODY for TO_LOOP): Likewise. * iconvdata/iso-2022-kr.c (BODY for TO_LOOP): Likewise. * iconvdata/johab.c (BODY for TO_LOOP): Likewise. * iconvdata/sjis.c (BODY for TO_LOOP): Likewise. * iconvdata/t.61.c (BODY for TO_LOOP): Likewise. * iconvdata/uhc.c (BODY for TO_LOOP): Likewise. * iconvdata/unicode.c (BODY for TO_LOOP): Likewise. * iconvdata/iso-2022-jp.c (TAG_none, TAG_language, TAG_language_j, TAG_language_ja, TAG_language_k, TAG_language_ko, TAG_language_z, TAG_language_zh, CURRENT_TAG_MASK): New enum values. (EMIT_SHIFT_TO_INIT): Don't emit an escape sequence if ASCII_set is already selected but set2 or tag are set. (conversion): New enum type. (cvlist_t): New type. (CVLIST, CVLIST_FIRST, CVLIST_REST): New macros. (conversion_lists): New array. (BODY for TO_LOOP): Keep track of Unicode 3.1 language tag. If "ja", prefer conversion to Japanese character sets. If "zh", prefer conversion to GB2312. If "ko", prefer conversion to KSC5601. Small optimizations. (INIT_PARAMS): Add tag. (UPDATE_PARAMS): Add tag. 2001-06-04 Bruno Haible <haible@clisp.cons.org> * locale/programs/locfile.c (write_locale_data): Before creat(), unlink the file, to avoid crashing the processes that mmap it. Change a double slash to a single slash. Free fname in case of error return. 2001-06-02 Jakub Jelinek <jakub@redhat.com> * sysdeps/i386/fpu/s_frexpl.S (__frexpl): Mostly revert 2000-12-03 changes, do the special handling for denormal numbers, not for normalized numbers (patch by <trevin@xmission.com>). * math/test-misc.c (main): Test frexpl with denormal arguments. 2001-06-04 Jakub Jelinek <jakub@redhat.com> * math/libm-test.inc (llround_test): Add two new llround tests. * sysdeps/ieee754/ldbl-96/s_llroundl.c (__llroundl): Don't allow overflow when rounding away from zero. 2001-06-04 Jakub Jelinek <jakub@redhat.com> * math/Makefile (libm-calls): Add e_log2, w_log2, remove s_log2. * math/math_private.h (__ieee754_log2, __ieee754_log2f, __ieee754_log2l): New prototypes. * sysdeps/generic/w_log2.c: New file. * sysdeps/generic/w_log2f.c: New file. * sysdeps/generic/w_log2l.c: New file. * sysdeps/generic/s_log2l.c: Move... * sysdeps/generic/e_log2l.c: ...to here. Rename to __ieee754_log2l. * sysdeps/ieee754/k_standard.c (__kernel_standard): Handle log2(0) and log2(x < 0). * sysdeps/i386/fpu/s_log2.S: Move... * sysdeps/i386/fpu/e_log2.S: ...to here. Rename to __ieee754_log2. * sysdeps/i386/fpu/s_log2f.S: Move... * sysdeps/i386/fpu/e_log2f.S: ...to here. Rename to __ieee754_log2f. * sysdeps/i386/fpu/s_log2l.S: Move... * sysdeps/i386/fpu/e_log2l.S: ...to here. Rename to __ieee754_log2l. * sysdeps/m68k/fpu/s_log2.S: Move... * sysdeps/m68k/fpu/e_log2.S: ...to here. Rename to __ieee754_log2. * sysdeps/m68k/fpu/s_log2f.S: Move... * sysdeps/m68k/fpu/e_log2f.S: ...to here. Rename to __ieee754_log2f. * sysdeps/m68k/fpu/s_log2l.S: Move... * sysdeps/m68k/fpu/e_log2l.S: ...to here. Rename to __ieee754_log2l. * sysdeps/ieee754/dbl-64/s_log2.c: Move... * sysdeps/ieee754/dbl-64/e_log2.c: ...to here. Rename to __ieee754_log2. * sysdeps/ieee754/flt-32/s_log2f.c: Move... * sysdeps/ieee754/flt-32/e_log2f.c: ...to here. Rename to __ieee754_log2f. 2001-06-04 Jakub Jelinek <jakub@redhat.com> * sysdeps/generic/w_exp2.c (u_threshold): Lower threshold so that even arguments which result in denormalized exp2 are accepted. (__exp2): Arguments equal to u_threshold already result into underflow. * sysdeps/generic/w_exp2f.c (u_threshold, __exp2f): Likewise. * sysdeps/generic/w_exp2l.c (u_threshold, __exp2l): Likewise. * sysdeps/ieee754/dbl-64/e_exp2.c (__ieee754_exp2): Lomark was too low, with corrected lowmark use greaterequal, not greater. * sysdeps/ieee754/flt-32/e_exp2f.c (__ieee754_exp2f): Likewise. 2001-06-04 Jakub Jelinek <jakub@redhat.com> * math/libm-test.inc (ilogb_test): Test that ilogb(+-Inf) == INT_MAX. * sysdeps/i386/fpu/s_ilogb.S (__ilogb): Return INT_MAX for +-Inf. * sysdeps/i386/fpu/s_ilogbf.S (__ilogbf): Likewise. * sysdeps/i386/fpu/s_ilogbl.S (__ilogbl): Likewise. * sysdeps/ieee754/dbl-64/s_ilogb.c (__ilogb): Likewise. * sysdeps/ieee754/flt-32/s_ilogbf.c (__ilogbf): Likewise. * sysdeps/ieee754/ldbl-128/s_ilogbl.c (__ilogbl): Likewise. * sysdeps/ieee754/ldbl-96/s_ilogbl.c (__ilogbl): Likewise. 2001-06-04 Jakub Jelinek <jakub@redhat.com> * sysdeps/generic/w_coshl.c (__coshl): Test if finite argument gave non-finite result instead of using constant in generic version. * sysdeps/generic/w_coshf.c (__coshf): Likewise. * sysdeps/generic/w_cosh.c (__cosh): Likewise. * sysdeps/generic/w_exp10.c (o_threshold, u_threshold): Remove. (__exp10): Test if finite argument gave non-finite result. * sysdeps/generic/w_exp10f.c (o_threshold, u_threshold, __exp10f): Likewise. * sysdeps/generic/w_exp10l.c (o_threshold, u_threshold, __exp10l): Likewise. 2001-06-04 Jakub Jelinek <jakub@redhat.com> * sysdeps/ieee754/ldbl-96/e_coshl.c (__ieee754_coshl): Fix overflow threshold constant (log(LDBL_MAX)+M_LN2l). 2001-05-29 Bruno Haible <haible@clisp.cons.org> * locale/programs/ld-ctype.c (idx_table): New struct type. (idx_table_init, idx_table_get, idx_table_add): New functions. (MAX_CHARNAMES_IDX): Remove macro. (locale_ctype_t): Change type of charnames_idx field. (ctype_startup): Change initialization of charnames_idx field. (find_idx): Use idx_table_get and idx_table_add for speed. * locale/programs/charmap.c (charmap_new_char): Fix ucs4 value computation of characters in a range. 2001-05-29 Bruno Haible <haible@clisp.cons.org> * iconvdata/gb18030.c (__fourbyte_to_ucs1): Add mappings for <U03F4>, <U03F5>. (__ucs_to_gb18030_tab1): Likewise. (BODY for FROM_LOOP): Add mapping for <U00010000>..<U0010FFFF>. (BODY for TO_LOOP): Likewise. * iconvdata/tst-table-charmap.sh: Update for charmaps containing <U00xxxxxx> syntax. * iconvdata/tst-table-from.c (bmp_only): New variable. (utf8_decode): If bmp_only, don't return characters outside Unicode plane 0. (main): When testing UTF-8 or GB18030, set bmp_only to 1. Don't print a conversion line if utf8_decode returns NULL. * iconvdata/tst-table-to.c (main): When testing encodings other than UTF-8 and GB18030, loop upto U+30000 instead of U+10000. Use UTF-8 instead of UCS-2 as input. * iconvdata/tst-table.sh: For GB18030, use only the part < 0x10000 of the charmap. 2001-05-29 Bruno Haible <haible@clisp.cons.org> * iconvdata/cns11643l1.c: Update to Unicode 3.1. (__cns11643l1_to_ucs4_tab): Regenerated. (__cns11643l1_from_ucs4_tab12): Regenerated. * iconvdata/cns11643.c: Update to Unicode 3.1. (__cns11643l14_to_ucs4_tab): Remove array. (__cns11643l3_to_ucs4_tab, __cns11643l4_to_ucs4_tab, __cns11643l5_to_ucs4_tab, __cns11643l6_to_ucs4_tab, __cns11643l7_to_ucs4_tab, __cns11643l15_to_ucs4_tab): New arrays. (__cns11643_from_ucs4p0_tab): Renamed from __cns11643_from_ucs4_tab. (__cns11643_from_ucs4p2_tab): New array. * iconvdata/cns11643.h (__cns11643l14_to_ucs4_tab): Remove declaration. (__cns11643l3_to_ucs4_tab, __cns11643l4_to_ucs4_tab, __cns11643l5_to_ucs4_tab, __cns11643l6_to_ucs4_tab, __cns11643l7_to_ucs4_tab, __cns11643l15_to_ucs4_tab): New declarations. (cns11643_to_ucs4): Treat planes 3, 4, 5, 6, 7, 15 instead of 14. (__cns11643_from_ucs4_tab): Remove declaration. (__cns11643_from_ucs4p0_tab, __cns11643_from_ucs4p2_tab): New declarations. (ucs4_to_cns11643): Update for new arrays. Treat U+3400..U+4DFF and U+20000..U+2A6D6. * iconvdata/cns11643l2.h (__cns11643_from_ucs4_tab): Remove declaration. (__cns11643_from_ucs4p0_tab): New declaration. (ucs4_to_cns11643l2): Update for new arrays. * iconvdata/iso-2022-cn-ext.c (BODY for FROM_LOOP): Handle planes 3 to 7. (BODY for TO_LOOP): Handle planes 3 to 7, instead of plane 14. * iconvdata/EUC-TW.irreversible: New file. * iconvdata/tst-table.sh: Use it. * iconvdata/Makefile (distribute): Add CP1255.irreversible, CP1258.irreversible, EUC-TW.irreversible. 2001-05-29 Bruno Haible <haible@clisp.cons.org> * locale/C-translit.h.in: Add transliterations for new Unicode 3.1 mathematical symbols.
Diffstat (limited to 'iconvdata')
37 files changed, 657 insertions, 388 deletions
diff --git a/iconvdata/8bit-gap.c b/iconvdata/8bit-gap.c index 7caa5427d5..e349270b8d 100644 --- a/iconvdata/8bit-gap.c +++ b/iconvdata/8bit-gap.c @@ -1,6 +1,6 @@ /* Generic conversion to and from 8bit charsets, converting from UCS using gaps. - Copyright (C) 1997, 1998, 1999, 2000 Free Software Foundation, Inc. + Copyright (C) 1997, 1998, 1999, 2000, 2001 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Ulrich Drepper <drepper@cygnus.com>, 1997. @@ -83,7 +83,10 @@ struct gap unsigned char res; \ \ if (__builtin_expect (ch, 0) >= 0xffff) \ - rp = NULL; \ + { \ + UNICODE_TAG_HANDLER (ch, 4); \ + rp = NULL; \ + } \ else \ while (ch > rp->end) \ ++rp; \ diff --git a/iconvdata/8bit-generic.c b/iconvdata/8bit-generic.c index bc05ccd8db..3a81080c4c 100644 --- a/iconvdata/8bit-generic.c +++ b/iconvdata/8bit-generic.c @@ -1,5 +1,5 @@ /* Generic conversion to and from 8bit charsets. - Copyright (C) 1997, 1998, 1999, 2000 Free Software Foundation, Inc. + Copyright (C) 1997, 1998, 1999, 2000, 2001 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Ulrich Drepper <drepper@cygnus.com>, 1997. @@ -67,6 +67,8 @@ if (__builtin_expect (ch >= sizeof (from_ucs4) / sizeof (from_ucs4[0]), 0)\ || (__builtin_expect (from_ucs4[ch], '\1') == '\0' && ch != 0)) \ { \ + UNICODE_TAG_HANDLER (ch, 4); \ + \ /* This is an illegal character. */ \ STANDARD_ERR_HANDLER (4); \ } \ diff --git a/iconvdata/ansi_x3.110.c b/iconvdata/ansi_x3.110.c index e1dd0ed0a7..a06a4f0658 100644 --- a/iconvdata/ansi_x3.110.c +++ b/iconvdata/ansi_x3.110.c @@ -1,5 +1,5 @@ /* Generic conversion to and from ANSI_X3.110-1983. - Copyright (C) 1998, 1999, 2000 Free Software Foundation, Inc. + Copyright (C) 1998, 1999, 2000, 2001 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Ulrich Drepper <drepper@cygnus.com>, 1998. @@ -549,6 +549,8 @@ static const char from_ucs4[][2] = cp = "\xd5"; \ else \ { \ + UNICODE_TAG_HANDLER (ch, 4); \ + \ /* Illegal characters. */ \ STANDARD_ERR_HANDLER (4); \ } \ diff --git a/iconvdata/big5.c b/iconvdata/big5.c index e06b44609e..4d1f1acca4 100644 --- a/iconvdata/big5.c +++ b/iconvdata/big5.c @@ -1,5 +1,5 @@ /* Mapping tables for Big5 handling. - Copyright (C) 1997, 1998, 1999, 2000 Free Software Foundation, Inc. + Copyright (C) 1997, 1998, 1999, 2000, 2001 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Ulrich Drepper <drepper@cygnus.com>, 1997. @@ -8564,6 +8564,7 @@ static const char from_ucs4_tab15[][2] = cp = from_ucs4_tab15[ch - 0xff01]; \ break; \ default: \ + UNICODE_TAG_HANDLER (ch, 4); \ /* Illegal character. */ \ cp = ""; \ break; \ diff --git a/iconvdata/big5hkscs.c b/iconvdata/big5hkscs.c index 7f4a8c3bf2..dff5456868 100644 --- a/iconvdata/big5hkscs.c +++ b/iconvdata/big5hkscs.c @@ -1,5 +1,5 @@ /* Mapping tables for Big5-HKSCS handling. - Copyright (C) 1997, 1998, 2000 Free Software Foundation, Inc. + Copyright (C) 1997, 1998, 2000, 2001 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Ulrich Drepper <drepper@cygnus.com>, 1997. Modified for Big5-HKSCS by Roger So <spacehunt@e-fever.org>, 2000. @@ -11772,6 +11772,8 @@ static struct \ if (cp[0] == '\0' && ch != 0) \ { \ + UNICODE_TAG_HANDLER (ch, 4); \ + \ /* Illegal character. */ \ STANDARD_ERR_HANDLER (4); \ } \ diff --git a/iconvdata/cp1255.c b/iconvdata/cp1255.c index 127f8be406..d81e7da85c 100644 --- a/iconvdata/cp1255.c +++ b/iconvdata/cp1255.c @@ -451,7 +451,10 @@ static const struct { else if (ch == 0x2122) \ res = 0x99; \ else \ - res = 0; \ + { \ + UNICODE_TAG_HANDLER (ch, 4); \ + res = 0; \ + } \ \ if (__builtin_expect (res != 0, 1)) \ { \ diff --git a/iconvdata/cp1258.c b/iconvdata/cp1258.c index b4168d9781..4868e7559c 100644 --- a/iconvdata/cp1258.c +++ b/iconvdata/cp1258.c @@ -777,7 +777,10 @@ static const struct { else if (ch == 0x2122) \ res = 0x99; \ else \ - res = 0; \ + { \ + UNICODE_TAG_HANDLER (ch, 4); \ + res = 0; \ + } \ \ if (__builtin_expect (res != 0, 1)) \ { \ diff --git a/iconvdata/euc-cn.c b/iconvdata/euc-cn.c index 58f47eda3f..29ed465979 100644 --- a/iconvdata/euc-cn.c +++ b/iconvdata/euc-cn.c @@ -1,5 +1,5 @@ /* Mapping tables for EUC-CN handling. - Copyright (C) 1998, 1999, 2000 Free Software Foundation, Inc. + Copyright (C) 1998, 1999, 2000, 2001 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Ulrich Drepper <drepper@cygnus.com>, 1998. @@ -140,6 +140,8 @@ { \ if (__builtin_expect (found, 0) == __UNKNOWN_10646_CHAR) \ { \ + UNICODE_TAG_HANDLER (ch, 4); \ + \ /* Illegal character. */ \ STANDARD_ERR_HANDLER (4); \ } \ diff --git a/iconvdata/euc-jp.c b/iconvdata/euc-jp.c index 8d8429614d..e0512cf12e 100644 --- a/iconvdata/euc-jp.c +++ b/iconvdata/euc-jp.c @@ -1,5 +1,5 @@ /* Mapping tables for EUC-JP handling. - Copyright (C) 1998, 1999, 2000 Free Software Foundation, Inc. + Copyright (C) 1998, 1999, 2000, 2001 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Ulrich Drepper <drepper@cygnus.com>, 1998. @@ -227,6 +227,8 @@ } \ else \ { \ + UNICODE_TAG_HANDLER (ch, 4); \ + \ /* Illegal character. */ \ STANDARD_ERR_HANDLER (4); \ } \ diff --git a/iconvdata/euc-kr.c b/iconvdata/euc-kr.c index 5ecdbb8182..7ec613b3f9 100644 --- a/iconvdata/euc-kr.c +++ b/iconvdata/euc-kr.c @@ -1,5 +1,5 @@ /* Mapping tables for EUC-KR handling. - Copyright (C) 1998, 1999, 2000 Free Software Foundation, Inc. + Copyright (C) 1998, 1999, 2000, 2001 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Jungshik Shin <jshin@pantheon.yale.edu> and Ulrich Drepper <drepper@cygnus.com>, 1998. @@ -142,10 +142,12 @@ euckr_from_ucs4 (uint32_t ch, unsigned char *cp) \ /* Decomposing Hangul syllables not available in KS C 5601 into \ Jamos should be considered either here or in euckr_from_ucs4() */ \ - euckr_from_ucs4 (ch, cp) ; \ + euckr_from_ucs4 (ch, cp); \ \ if (__builtin_expect (cp[0], '\1') == '\0' && ch != 0) \ { \ + UNICODE_TAG_HANDLER (ch, 4); \ + \ /* Illegal character. */ \ STANDARD_ERR_HANDLER (4); \ } \ diff --git a/iconvdata/euc-tw.c b/iconvdata/euc-tw.c index 39c2f1f7da..68a80f7c6d 100644 --- a/iconvdata/euc-tw.c +++ b/iconvdata/euc-tw.c @@ -1,5 +1,5 @@ /* Mapping tables for EUC-TW handling. - Copyright (C) 1998, 1999, 2000 Free Software Foundation, Inc. + Copyright (C) 1998, 1999, 2000, 2001 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Ulrich Drepper <drepper@cygnus.com>, 1998. @@ -61,14 +61,14 @@ } \ else \ { \ - /* Two or more byte character. First test whether the next \ - character is also available. */ \ + /* Two or more byte character. First test whether the next byte \ + is also available. */ \ uint32_t ch2; \ \ if (inptr + 1 >= inend) \ { \ - /* The second character is not available. Store the \ - intermediate result. */ \ + /* The second byte is not available. Store the intermediate \ + result. */ \ result = __GCONV_INCOMPLETE_INPUT; \ break; \ } \ @@ -99,7 +99,7 @@ \ if (ch == 0) \ { \ - /* The third or fourth character is not available. Store \ + /* The third or fourth byte is not available. Store \ the intermediate result. */ \ result = __GCONV_INCOMPLETE_INPUT; \ break; \ @@ -170,7 +170,7 @@ *outptr++ = ch; \ else \ { \ - /* Try the JIS character sets. */ \ + /* Try the CNS 11643 planes. */ \ size_t found; \ \ found = ucs4_to_cns11643l1 (ch, outptr, outend - outptr); \ @@ -199,6 +199,8 @@ } \ if (__builtin_expect (found, 0) == __UNKNOWN_10646_CHAR) \ { \ + UNICODE_TAG_HANDLER (ch, 4); \ + \ /* Illegal character. */ \ STANDARD_ERR_HANDLER (4); \ } \ diff --git a/iconvdata/gb18030.c b/iconvdata/gb18030.c index 8b8c472074..e3e9e8a66d 100644 --- a/iconvdata/gb18030.c +++ b/iconvdata/gb18030.c @@ -5683,7 +5683,8 @@ static const uint16_t __fourbyte_to_ucs1[] = [0x0309] = 0x03e5, [0x030a] = 0x03e6, [0x030b] = 0x03e7, [0x030c] = 0x03e8, [0x030d] = 0x03e9, [0x030e] = 0x03ea, [0x030f] = 0x03eb, [0x0310] = 0x03ec, [0x0311] = 0x03ed, [0x0312] = 0x03ee, [0x0313] = 0x03ef, [0x0314] = 0x03f0, - [0x0315] = 0x03f1, [0x0316] = 0x03f2, [0x0317] = 0x03f3, [0x0324] = 0x0400, + [0x0315] = 0x03f1, [0x0316] = 0x03f2, [0x0317] = 0x03f3, [0x0318] = 0x03f4, + [0x0319] = 0x03f5, [0x0324] = 0x0400, [0x0325] = 0x0402, [0x0326] = 0x0403, [0x0327] = 0x0404, [0x0328] = 0x0405, [0x0329] = 0x0406, [0x032a] = 0x0407, [0x032b] = 0x0408, [0x032c] = 0x0409, [0x032d] = 0x040a, [0x032e] = 0x040b, [0x032f] = 0x040c, [0x0330] = 0x040d, @@ -9517,6 +9518,7 @@ static const unsigned char __ucs_to_gb18030_tab1[][2] = [0x036b] = "\x23\x0f", [0x036c] = "\x23\x10", [0x036d] = "\x23\x11", [0x036e] = "\x23\x12", [0x036f] = "\x23\x13", [0x0370] = "\x23\x14", [0x0371] = "\x23\x15", [0x0372] = "\x23\x16", [0x0373] = "\x23\x17", + [0x0374] = "\x23\x18", [0x0375] = "\x23\x19", [0x0380] = "\x23\x24", [0x0381] = "\xa7\xa7", [0x0382] = "\x23\x25", [0x0383] = "\x23\x26", [0x0384] = "\x23\x27", [0x0385] = "\x23\x28", [0x0386] = "\x23\x29", [0x0387] = "\x23\x2a", [0x0388] = "\x23\x2b", @@ -21954,15 +21956,15 @@ static const unsigned char __ucs_to_gb18030_tab2[][2] = } \ else \ { \ - /* Two or four byte character. First test whether the \ - next character is also available. */ \ + /* Two or four byte character. First test whether the next \ + byte is also available. */ \ uint32_t ch2; \ unsigned long int idx; \ \ if (inptr + 1 >= inend) \ { \ - /* The second character is not available. Store \ - the intermediate result. */ \ + /* The second byte is not available. Store the \ + intermediate result. */ \ result = __GCONV_INCOMPLETE_INPUT; \ break; \ } \ @@ -21992,7 +21994,7 @@ static const unsigned char __ucs_to_gb18030_tab2[][2] = \ if (inptr + 3 >= inend) \ { \ - /* Not all characters are available. Store \ + /* Not all bytes are available. Store \ the intermediate result. */ \ result = __GCONV_INCOMPLETE_INPUT; \ break; \ @@ -22035,17 +22037,19 @@ static const unsigned char __ucs_to_gb18030_tab2[][2] = + ch3 - 0x81) * 10 + ch4 - 0x30; \ \ if (idx <= 0x4A18) \ - ch = __fourbyte_to_ucs1[idx]; \ + ch = __fourbyte_to_ucs1[idx]; \ else if (idx >= 0x4ABD && idx <= 0x4F49) \ - ch = idx + 0x5543; \ + ch = idx + 0x5543; \ else if (idx >= 0x4F4D && idx <= 0x4F83) \ - ch = __fourbyte_to_ucs2[idx-0x4F4D]; \ + ch = __fourbyte_to_ucs2[idx-0x4F4D]; \ else if (idx >= 0x56BD && idx <= 0x8260) \ - ch = idx + 0x5543; \ + ch = idx + 0x5543; \ else if (idx >= 0x93A9 && idx <= 0x99F9) \ - ch = __fourbyte_to_ucs3[idx-0x93A9]; \ + ch = __fourbyte_to_ucs3[idx-0x93A9]; \ + else if (idx >= 0x2E248 && idx <= 0x12E247) \ + ch = idx - 0x1E248; \ else \ - ch = 0; \ + ch = 0; \ \ if (ch == 0 && *inptr != '\0') \ { \ @@ -22148,6 +22152,11 @@ static const unsigned char __ucs_to_gb18030_tab2[][2] = len = 4; \ } \ } \ + else if (ch >= 0x10000 && ch <= 0x10FFFF) \ + { \ + idx = ch + 0x1E248; \ + len = 4; \ + } \ else \ len = 0; \ \ @@ -22175,7 +22184,7 @@ static const unsigned char __ucs_to_gb18030_tab2[][2] = } \ else /* len == 4 */ \ { \ - /* See whether there is enough room for the second byte we \ + /* See whether there is enough room for all four bytes we \ write. */ \ if (__builtin_expect (outptr + 3 >= outend, 0)) \ { \ diff --git a/iconvdata/gbk.c b/iconvdata/gbk.c index b07f39294a..23e7129740 100644 --- a/iconvdata/gbk.c +++ b/iconvdata/gbk.c @@ -1,5 +1,5 @@ /* Mapping tables for GBK handling. - Copyright (C) 1999, 2000 Free Software Foundation, Inc. + Copyright (C) 1999, 2000, 2001 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Sean Chen <sean.chen@turbolinux.com>, 1999. @@ -13471,6 +13471,7 @@ static const char __gbk_from_ucs4_tab12[][2] = cp = "\xa1\xe9\0\0\xa1\xea\0\0\xa9\x56\0\0\xa3\xfe\0\0\xa9\x57\0\0\xa3\xa4" + ((ch - 0xffe0) * 4); \ break; \ default: \ + UNICODE_TAG_HANDLER (ch, 4); \ cp = ""; \ break; \ } \ diff --git a/iconvdata/ibm930.c b/iconvdata/ibm930.c index 0d09b7bec2..406f3f5174 100644 --- a/iconvdata/ibm930.c +++ b/iconvdata/ibm930.c @@ -203,6 +203,8 @@ enum \ if (__builtin_expect (ch, 0) >= 0xffff) \ { \ + UNICODE_TAG_HANDLER (ch, 4); \ + \ if (! ignore_errors_p ()) \ { \ result = __GCONV_ILLEGAL_INPUT; \ diff --git a/iconvdata/ibm932.c b/iconvdata/ibm932.c index 07fdd25f6e..d6c5877c0e 100644 --- a/iconvdata/ibm932.c +++ b/iconvdata/ibm932.c @@ -1,5 +1,5 @@ /* Conversion from and to IBM932. - Copyright (C) 2000 Free Software Foundation, Inc. + Copyright (C) 2000, 2001 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Masahide Washizawa <washi@jp.ibm.com>, 2000. @@ -148,7 +148,7 @@ { \ const struct gap *rp = __ucs4_to_ibm932sb_idx; \ unsigned char sc; \ - uint32_t ch = get32(inptr); \ + uint32_t ch = get32 (inptr); \ uint16_t found = TRUE; \ uint32_t i; \ uint32_t low; \ @@ -156,7 +156,10 @@ uint16_t pccode; \ \ if (__builtin_expect (ch, 0) >= 0xffff) \ + { \ + UNICODE_TAG_HANDLER (ch, 4); \ rp = NULL; \ + } \ else \ while (ch > rp->end) \ ++rp; \ @@ -188,26 +191,26 @@ break; \ } \ } \ - if (found) \ - { \ - if (__builtin_expect (outptr + 2 > outend, 0)) \ - { \ - result = __GCONV_FULL_OUTPUT; \ - break; \ - } \ - *outptr++ = pccode >> 8 & 0xff; \ - *outptr++ = pccode & 0xff; \ - } \ - else \ - { \ - /* This is an illegal character. */ \ - if (! ignore_errors_p ()) \ - { \ - result = __GCONV_ILLEGAL_INPUT; \ - break; \ - } \ - ++*irreversible; \ - } \ + if (found) \ + { \ + if (__builtin_expect (outptr + 2 > outend, 0)) \ + { \ + result = __GCONV_FULL_OUTPUT; \ + break; \ + } \ + *outptr++ = pccode >> 8 & 0xff; \ + *outptr++ = pccode & 0xff; \ + } \ + else \ + { \ + /* This is an illegal character. */ \ + if (! ignore_errors_p ()) \ + { \ + result = __GCONV_ILLEGAL_INPUT; \ + break; \ + } \ + ++*irreversible; \ + } \ } \ else \ { \ @@ -224,8 +227,8 @@ *outptr++ = sc; \ } \ \ - /* Now that we wrote the output increment the input pointer. */ \ - inptr += 4; \ + /* Now that we wrote the output increment the input pointer. */ \ + inptr += 4; \ } #define LOOP_NEED_FLAGS #include <iconv/loop.c> diff --git a/iconvdata/ibm933.c b/iconvdata/ibm933.c index 92a9d5b52a..9d9d76aee8 100644 --- a/iconvdata/ibm933.c +++ b/iconvdata/ibm933.c @@ -205,6 +205,8 @@ enum \ if (__builtin_expect (ch, 0) >= 0xffff) \ { \ + UNICODE_TAG_HANDLER (ch, 4); \ + \ if (! ignore_errors_p ()) \ { \ result = __GCONV_ILLEGAL_INPUT; \ diff --git a/iconvdata/ibm935.c b/iconvdata/ibm935.c index 9728894b0b..91e3255baf 100644 --- a/iconvdata/ibm935.c +++ b/iconvdata/ibm935.c @@ -203,6 +203,8 @@ enum \ if (__builtin_expect (ch, 0) >= 0xffff) \ { \ + UNICODE_TAG_HANDLER (ch, 4); \ + \ if (! ignore_errors_p ()) \ { \ result = __GCONV_ILLEGAL_INPUT; \ diff --git a/iconvdata/ibm937.c b/iconvdata/ibm937.c index 3dca817914..e47ff941ae 100644 --- a/iconvdata/ibm937.c +++ b/iconvdata/ibm937.c @@ -195,6 +195,8 @@ enum / sizeof (__ucs4_to_ibm937db[0])), 0) \ || __builtin_expect (cp[0], '\1') == '\0') \ { \ + UNICODE_TAG_HANDLER (ch, 4); \ + \ /* This is an illegal character. */ \ if (! ignore_errors_p ()) \ { \ diff --git a/iconvdata/ibm939.c b/iconvdata/ibm939.c index 02307a0084..77a2b2b8a9 100644 --- a/iconvdata/ibm939.c +++ b/iconvdata/ibm939.c @@ -203,6 +203,7 @@ enum \ if (__builtin_expect (ch, 0) >= 0xffff) \ { \ + UNICODE_TAG_HANDLER (ch, 4); \ rp1 = NULL; \ rp2 = NULL; \ } \ diff --git a/iconvdata/ibm943.c b/iconvdata/ibm943.c index 5ff27f558a..3a742152a8 100644 --- a/iconvdata/ibm943.c +++ b/iconvdata/ibm943.c @@ -1,5 +1,5 @@ /* Conversion from and to IBM943. - Copyright (C) 2000 Free Software Foundation, Inc. + Copyright (C) 2000, 2001 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Masahide Washizawa <washi@jp.ibm.com>, 2000. @@ -156,7 +156,10 @@ uint16_t pccode; \ \ if (__builtin_expect (ch, 0) >= 0xffff) \ + { \ + UNICODE_TAG_HANDLER (ch, 4); \ rp = NULL; \ + } \ else \ while (ch > rp->end) \ ++rp; \ @@ -188,26 +191,26 @@ break; \ } \ } \ - if (found) \ - { \ - if (__builtin_expect (outptr + 2 > outend, 0)) \ - { \ - result = __GCONV_FULL_OUTPUT; \ - break; \ - } \ - *outptr++ = pccode >> 8 & 0xff; \ - *outptr++ = pccode & 0xff; \ - } \ - else \ - { \ - /* This is an illegal character. */ \ - if (! ignore_errors_p ()) \ - { \ - result = __GCONV_ILLEGAL_INPUT; \ - break; \ - } \ - ++*irreversible; \ - } \ + if (found) \ + { \ + if (__builtin_expect (outptr + 2 > outend, 0)) \ + { \ + result = __GCONV_FULL_OUTPUT; \ + break; \ + } \ + *outptr++ = pccode >> 8 & 0xff; \ + *outptr++ = pccode & 0xff; \ + } \ + else \ + { \ + /* This is an illegal character. */ \ + if (! ignore_errors_p ()) \ + { \ + result = __GCONV_ILLEGAL_INPUT; \ + break; \ + } \ + ++*irreversible; \ + } \ } \ else \ { \ @@ -224,8 +227,8 @@ *outptr++ = sc; \ } \ \ - /* Now that we wrote the output increment the input pointer. */ \ - inptr += 4; \ + /* Now that we wrote the output increment the input pointer. */ \ + inptr += 4; \ } #define LOOP_NEED_FLAGS #include <iconv/loop.c> diff --git a/iconvdata/iso-2022-cn-ext.c b/iconvdata/iso-2022-cn-ext.c index 9fd1d64b7d..aeb921a43c 100644 --- a/iconvdata/iso-2022-cn-ext.c +++ b/iconvdata/iso-2022-cn-ext.c @@ -538,6 +538,7 @@ enum } \ if (used == ASCII_set) \ { \ + UNICODE_TAG_HANDLER (ch, 4); \ STANDARD_ERR_HANDLER (4); \ } \ } \ diff --git a/iconvdata/iso-2022-cn.c b/iconvdata/iso-2022-cn.c index 6eeeea3606..61f4bdda0c 100644 --- a/iconvdata/iso-2022-cn.c +++ b/iconvdata/iso-2022-cn.c @@ -1,5 +1,5 @@ /* Conversion module for ISO-2022-CN. - Copyright (C) 1999, 2000 Free Software Foundation, Inc. + Copyright (C) 1999, 2000, 2001 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Ulrich Drepper <drepper@cygnus.com>, 1999. @@ -320,6 +320,8 @@ enum used = GB2312_set + CNS11643_1_set - used; \ else \ { \ + UNICODE_TAG_HANDLER (ch, 4); \ + \ /* Even this does not work. Error. */ \ STANDARD_ERR_HANDLER (4); \ } \ diff --git a/iconvdata/iso-2022-jp.c b/iconvdata/iso-2022-jp.c index 8825dcc0fd..4fba9d993f 100644 --- a/iconvdata/iso-2022-jp.c +++ b/iconvdata/iso-2022-jp.c @@ -1,5 +1,5 @@ -/* Conversion module for ISO-2022-JP. - Copyright (C) 1998, 1999, 2000 Free Software Foundation, Inc. +/* Conversion module for ISO-2022-JP and ISO-2022-JP-2. + Copyright (C) 1998, 1999, 2000, 2001 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Ulrich Drepper <drepper@cygnus.com>, 1998. @@ -90,14 +90,14 @@ struct iso2022jp_data enum { ASCII_set = 0, - JISX0208_1978_set = 8, - JISX0208_1983_set = 16, - JISX0201_Roman_set = 24, - JISX0201_Kana_set = 32, - GB2312_set = 40, - KSC5601_set = 48, - JISX0212_set = 56, - CURRENT_SEL_MASK = 56 + JISX0208_1978_set = 1 << 3, + JISX0208_1983_set = 2 << 3, + JISX0201_Roman_set = 3 << 3, + JISX0201_Kana_set = 4 << 3, + GB2312_set = 5 << 3, + KSC5601_set = 6 << 3, + JISX0212_set = 7 << 3, + CURRENT_SEL_MASK = 7 << 3 }; /* The second value stored is the designation of the G2 set. The following @@ -105,9 +105,25 @@ enum enum { UNSPECIFIED_set = 0, - ISO88591_set = 64, - ISO88597_set = 128, - CURRENT_ASSIGN_MASK = 192 + ISO88591_set = 1 << 6, + ISO88597_set = 2 << 6, + CURRENT_ASSIGN_MASK = 3 << 6 +}; + +/* The third value, only used during conversion from Unicode to ISO-2022-JP-2, + describes the language tag parsing status. The possible values are as + follows. Values >= TAG_language are temporary tag parsing states. */ +enum +{ + TAG_none = 0, + TAG_language = 4 << 8, + TAG_language_j = 5 << 8, + TAG_language_ja = 1 << 8, + TAG_language_k = 6 << 8, + TAG_language_ko = 2 << 8, + TAG_language_z = 7 << 8, + TAG_language_zh = 3 << 8, + CURRENT_TAG_MASK = 7 << 8 }; @@ -198,7 +214,8 @@ gconv_end (struct __gconv_step *data) \ if ((data->__statep->__count & ~7) != ASCII_set) \ { \ - if (dir == from_iso2022jp) \ + if (dir == from_iso2022jp \ + || (data->__statep->__count & CURRENT_SEL_MASK) == ASCII_set) \ { \ /* It's easy, we don't have to emit anything, we just reset the \ state for the input. Note that this also clears the G2 \ @@ -503,6 +520,23 @@ gconv_end (struct __gconv_step *data) /* Next, define the other direction. */ + +enum conversion { none = 0, european, japanese, chinese, korean, other }; + +/* A datatype for conversion lists. */ +typedef unsigned int cvlist_t; +#define CVLIST(cv1, cv2, cv3, cv4, cv5) \ + ((cv1) + ((cv2) << 3) + ((cv3) << 6) + ((cv4) << 9) + ((cv5) << 12)) +#define CVLIST_FIRST(cvl) ((cvl) & ((1 << 3) - 1)) +#define CVLIST_REST(cvl) ((cvl) >> 3) +static const cvlist_t conversion_lists[4] = + { + /* TAG_none */ CVLIST (japanese, european, chinese, korean, other), + /* TAG_language_ja */ CVLIST (japanese, european, chinese, korean, other), + /* TAG_language_ko */ CVLIST (korean, european, japanese, chinese, other), + /* TAG_language_zh */ CVLIST (chinese, european, japanese, korean, other) + }; + #define MIN_NEEDED_INPUT MIN_NEEDED_TO #define MIN_NEEDED_OUTPUT MIN_NEEDED_FROM #define MAX_NEEDED_OUTPUT (MAX_NEEDED_FROM + 2) @@ -510,12 +544,56 @@ gconv_end (struct __gconv_step *data) #define BODY \ { \ uint32_t ch; \ - size_t written = 0; \ + size_t written; \ \ ch = get32 (inptr); \ \ + if (var == iso2022jp2) \ + { \ + /* Handle Unicode tag characters (range U+E0000..U+E007F). */ \ + if (__builtin_expect ((ch >> 7) == (0xe0000 >> 7), 0)) \ + { \ + ch &= 0x7f; \ + if (ch >= 'A' && ch <= 'Z') \ + ch += 'a' - 'A'; \ + if (ch == 0x01) \ + tag = TAG_language; \ + else if (ch == 'j' && tag == TAG_language) \ + tag = TAG_language_j; \ + else if (ch == 'a' && tag == TAG_language_j) \ + tag = TAG_language_ja; \ + else if (ch == 'k' && tag == TAG_language) \ + tag = TAG_language_k; \ + else if (ch == 'o' && tag == TAG_language_k) \ + tag = TAG_language_ko; \ + else if (ch == 'z' && tag == TAG_language) \ + tag = TAG_language_z; \ + else if (ch == 'h' && tag == TAG_language_z) \ + tag = TAG_language_zh; \ + else if (ch == 0x7f) \ + tag = TAG_none; \ + else \ + { \ + /* Other tag characters reset the tag parsing state (if the \ + current state is a temporary state) or are ignored (if \ + the current state is a stable one). */ \ + if (tag >= TAG_language) \ + tag = TAG_none; \ + } \ + \ + inptr += 4; \ + continue; \ + } \ + \ + /* Non-tag characters reset the tag parsing state, if the current \ + state is a temporary state. */ \ + if (__builtin_expect (tag >= TAG_language, 0)) \ + tag = TAG_none; \ + } \ + \ /* First see whether we can write the character using the currently \ - selected character set. */ \ + selected character set. But ignore the selected character set if \ + the current language tag shows different preferences. */ \ if (set == ASCII_set) \ { \ /* Please note that the NUL byte is *not* matched if we are not \ @@ -525,58 +603,75 @@ gconv_end (struct __gconv_step *data) { \ *outptr++ = ch; \ written = 1; \ + \ + /* At the beginning of a line, G2 designation is cleared. */ \ + if (var == iso2022jp2 && ch == 0x0a) \ + set2 = UNSPECIFIED_set; \ } \ - /* At the beginning of a line, G2 designation is cleared. */ \ - if (var == iso2022jp2 && ch == 0x0a) \ - set2 = UNSPECIFIED_set; \ + else \ + written = __UNKNOWN_10646_CHAR; \ } \ /* ISO-2022-JP recommends to encode the newline character always in \ ASCII since this allows a context-free interpretation of the \ characters at the beginning of the next line. Otherwise it would \ have to be known whether the last line ended using ASCII or \ JIS X 0201. */ \ - else if (set == JISX0201_Roman_set) \ + else if (set == JISX0201_Roman_set \ + && (__builtin_expect (tag == TAG_none, 1) \ + || tag == TAG_language_ja)) \ { \ - unsigned char buf[2]; \ + unsigned char buf[1]; \ written = ucs4_to_jisx0201 (ch, buf); \ - if (written != __UNKNOWN_10646_CHAR && buf[0] > 0x20 \ - && buf[0] < 0x80) \ + if (written != __UNKNOWN_10646_CHAR) \ { \ - *outptr++ = buf[0]; \ - written = 1; \ + if (buf[0] > 0x20 && buf[0] < 0x80) \ + { \ + *outptr++ = buf[0]; \ + written = 1; \ + } \ + else \ + written = __UNKNOWN_10646_CHAR; \ } \ - else \ - written = __UNKNOWN_10646_CHAR; \ } \ - else if (set == JISX0201_Kana_set) \ + else if (set == JISX0201_Kana_set \ + && (__builtin_expect (tag == TAG_none, 1) \ + || tag == TAG_language_ja)) \ { \ - unsigned char buf[2]; \ + unsigned char buf[1]; \ written = ucs4_to_jisx0201 (ch, buf); \ - if (written != __UNKNOWN_10646_CHAR && buf[0] > 0xa0 \ - && buf[0] < 0xe0) \ + if (written != __UNKNOWN_10646_CHAR) \ { \ - *outptr++ = buf[0] - 0x80; \ - written = 1; \ + if (buf[0] > 0xa0 && buf[0] < 0xe0) \ + { \ + *outptr++ = buf[0] - 0x80; \ + written = 1; \ + } \ + else \ + written = __UNKNOWN_10646_CHAR; \ } \ - else \ - written = __UNKNOWN_10646_CHAR; \ } \ else \ { \ - if (set == JISX0208_1978_set || set == JISX0208_1983_set) \ + if ((set == JISX0208_1978_set || set == JISX0208_1983_set) \ + && (__builtin_expect (tag == TAG_none, 1) \ + || tag == TAG_language_ja)) \ written = ucs4_to_jisx0208 (ch, outptr, outend - outptr); \ - else if (set == JISX0212_set) \ + else if (set == JISX0212_set \ + && (__builtin_expect (tag == TAG_none, 1) \ + || tag == TAG_language_ja)) \ written = ucs4_to_jisx0212 (ch, outptr, outend - outptr); \ - else if (set == GB2312_set) \ + else if (set == GB2312_set \ + && (__builtin_expect (tag == TAG_none, 1) \ + || tag == TAG_language_zh)) \ written = ucs4_to_gb2312 (ch, outptr, outend - outptr); \ + else if (set == KSC5601_set \ + && (__builtin_expect (tag == TAG_none, 1) \ + || tag == TAG_language_ko)) \ + written = ucs4_to_ksc5601 (ch, outptr, outend - outptr); \ else \ - { \ - assert (set == KSC5601_set); \ - \ - written = ucs4_to_ksc5601 (ch, outptr, outend - outptr); \ - } \ + written = __UNKNOWN_10646_CHAR; \ \ - if (__builtin_expect (written, 1) == 0) \ + if (__builtin_expect (written == 0, 0)) \ { \ result = __GCONV_FULL_OUTPUT; \ break; \ @@ -585,18 +680,19 @@ gconv_end (struct __gconv_step *data) outptr += written; \ } \ \ - if (written == __UNKNOWN_10646_CHAR || written == 0) \ + if (written == __UNKNOWN_10646_CHAR \ + && __builtin_expect (tag == TAG_none, 1)) \ { \ if (set2 == ISO88591_set) \ { \ - if (__builtin_expect (outptr + 3 > outend, 0)) \ + if (ch >= 0x80 && ch <= 0xff) \ { \ - result = __GCONV_FULL_OUTPUT; \ - break; \ - } \ + if (__builtin_expect (outptr + 3 > outend, 0)) \ + { \ + result = __GCONV_FULL_OUTPUT; \ + break; \ + } \ \ - if (ch >= 0x80 && ch <= 0xff) \ - { \ *outptr++ = ESC; \ *outptr++ = 'N'; \ *outptr++ = ch & 0x7f; \ @@ -629,18 +725,15 @@ gconv_end (struct __gconv_step *data) } \ } \ \ - if (written == __UNKNOWN_10646_CHAR || written == 0) \ + if (written == __UNKNOWN_10646_CHAR) \ { \ - /* Either this is an unknown character or we have to switch \ - the currently selected character set. The character sets \ - do not code entirely separate parts of ISO 10646 and \ - therefore there is no single correct result. If we choose \ - the character set to use wrong we might be end up with \ - using yet another character set for the next character \ - though the current and the next could be encoded with one \ - character set. We leave this kind of optimization for \ - later and now simply use a fixed order in which we test for \ - availability */ \ + /* The attempts to use the currently selected character set \ + failed, either because the language tag changed, or because \ + the character requires a different character set, or because \ + the character is unknown. \ + The CJK character sets partially overlap when seen as subsets \ + of ISO 10646; therefore there is no single correct result. \ + We use a preferrence order which depends on the language tag. */ \ \ if (ch <= 0x7f) \ { \ @@ -671,235 +764,298 @@ gconv_end (struct __gconv_step *data) else \ { \ /* Now it becomes difficult. We must search the other \ - character sets one by one and we cannot use simple \ - arithmetic to determine whether the character can be \ - encoded using this set. */ \ - size_t written; \ + character sets one by one. Use an ordered conversion \ + list that depends on the current language tag. */ \ + cvlist_t conversion_list; \ unsigned char buf[2]; \ \ - written = ucs4_to_jisx0201 (ch, buf); \ - if (written != __UNKNOWN_10646_CHAR && buf[0] < 0x80) \ - { \ - /* We use JIS X 0201. */ \ - if (__builtin_expect (outptr + 3 > outend, 0)) \ - { \ - result = __GCONV_FULL_OUTPUT; \ - break; \ - } \ - \ - *outptr++ = ESC; \ - *outptr++ = '('; \ - *outptr++ = 'J'; \ - set = JISX0201_Roman_set; \ + result = __GCONV_ILLEGAL_INPUT; \ \ - if (__builtin_expect (outptr + 1 > outend, 0)) \ - { \ - result = __GCONV_FULL_OUTPUT; \ - break; \ - } \ - *outptr++ = buf[0]; \ - } \ + if (var == iso2022jp2) \ + conversion_list = conversion_lists[tag >> 8]; \ else \ - { \ - written = ucs4_to_jisx0208 (ch, buf, 2); \ - if (written != __UNKNOWN_10646_CHAR) \ - { \ - /* We use JIS X 0208. */ \ - if (__builtin_expect (outptr + 3 > outend, 0)) \ - { \ - result = __GCONV_FULL_OUTPUT; \ - break; \ - } \ + conversion_list = CVLIST (japanese, 0, 0, 0, 0); \ \ - *outptr++ = ESC; \ - *outptr++ = '$'; \ - *outptr++ = 'B'; \ - set = JISX0208_1983_set; \ + do \ + switch (CVLIST_FIRST (conversion_list)) \ + { \ + case european: \ \ - if (__builtin_expect (outptr + 2 > outend, 0)) \ - { \ - result = __GCONV_FULL_OUTPUT; \ - break; \ - } \ - *outptr++ = buf[0]; \ - *outptr++ = buf[1]; \ - } \ - else if (__builtin_expect (var, iso2022jp2) == iso2022jp) \ - { \ - /* We have no other choice. */ \ - STANDARD_ERR_HANDLER (4); \ - } \ - else \ - { \ - written = ucs4_to_jisx0212 (ch, buf, 2); \ - if (written != __UNKNOWN_10646_CHAR) \ - { \ - /* We use JIS X 0212. */ \ - if (__builtin_expect (outptr + 4 > outend, 0)) \ - { \ - result = __GCONV_FULL_OUTPUT; \ - break; \ - } \ - *outptr++ = ESC; \ - *outptr++ = '$'; \ - *outptr++ = '('; \ - *outptr++ = 'D'; \ - set = JISX0212_set; \ - \ - if (__builtin_expect (outptr + 2 > outend, 0)) \ - { \ - result = __GCONV_FULL_OUTPUT; \ - break; \ - } \ - *outptr++ = buf[0]; \ - *outptr++ = buf[1]; \ - } \ - else \ - { \ - written = ucs4_to_jisx0201 (ch, buf); \ - if (written != __UNKNOWN_10646_CHAR \ - && buf[0] >= 0x80) \ - { \ - /* We use JIS X 0201. */ \ - if (__builtin_expect (outptr + 3 > outend, 0)) \ - { \ - result = __GCONV_FULL_OUTPUT; \ - break; \ - } \ + /* Try ISO 8859-1 upper half. */ \ + if (ch >= 0x80 && ch <= 0xff) \ + { \ + if (set2 != ISO88591_set) \ + { \ + if (__builtin_expect (outptr + 3 > outend, 0)) \ + { \ + result = __GCONV_FULL_OUTPUT; \ + break; \ + } \ + *outptr++ = ESC; \ + *outptr++ = '.'; \ + *outptr++ = 'A'; \ + set2 = ISO88591_set; \ + } \ \ - *outptr++ = ESC; \ - *outptr++ = '('; \ - *outptr++ = 'I'; \ - set = JISX0201_Kana_set; \ + if (__builtin_expect (outptr + 3 > outend, 0)) \ + { \ + result = __GCONV_FULL_OUTPUT; \ + break; \ + } \ + *outptr++ = ESC; \ + *outptr++ = 'N'; \ + *outptr++ = ch - 0x80; \ + result = __GCONV_OK; \ + break; \ + } \ \ - if (__builtin_expect (outptr + 1 > outend, 0)) \ - { \ - result = __GCONV_FULL_OUTPUT; \ - break; \ - } \ - *outptr++ = buf[0] - 0x80; \ - } \ - else if (ch != 0xa5 && ch >= 0x80 && ch <= 0xff) \ - { \ - /* ISO 8859-1 upper half. */ \ - if (__builtin_expect (outptr + 3 > outend, 0)) \ - { \ - result = __GCONV_FULL_OUTPUT; \ - break; \ - } \ - \ - *outptr++ = ESC; \ - *outptr++ = '.'; \ - *outptr++ = 'A'; \ - set2 = ISO88591_set; \ + /* Try ISO 8859-7 upper half. */ \ + { \ + const struct gap *rp = from_idx; \ \ - if (__builtin_expect (outptr + 3 > outend, 0)) \ - { \ - result = __GCONV_FULL_OUTPUT; \ - break; \ - } \ - *outptr++ = ESC; \ - *outptr++ = 'N'; \ - *outptr++ = ch - 0x80; \ - } \ - else \ + while (ch > rp->end) \ + ++rp; \ + if (ch >= rp->start) \ + { \ + unsigned char res = \ + iso88597_from_ucs4[ch - 0xa0 + rp->idx]; \ + if (res != '\0') \ { \ - written = ucs4_to_gb2312 (ch, buf, 2); \ - if (written != __UNKNOWN_10646_CHAR) \ + if (set2 != ISO88597_set) \ { \ - /* We use GB 2312. */ \ if (__builtin_expect (outptr + 3 > outend, 0))\ { \ result = __GCONV_FULL_OUTPUT; \ break; \ } \ - \ *outptr++ = ESC; \ - *outptr++ = '$'; \ - *outptr++ = 'A'; \ - set = GB2312_set; \ - \ - if (__builtin_expect (outptr + 2 > outend, 0))\ - { \ - result = __GCONV_FULL_OUTPUT; \ - break; \ - } \ - *outptr++ = buf[0]; \ - *outptr++ = buf[1]; \ + *outptr++ = '.'; \ + *outptr++ = 'F'; \ + set2 = ISO88597_set; \ } \ - else \ + \ + if (__builtin_expect (outptr + 3 > outend, 0)) \ { \ - written = ucs4_to_ksc5601 (ch, buf, 2); \ - if (written != __UNKNOWN_10646_CHAR) \ - { \ - /* We use KSC 5601. */ \ - if (__builtin_expect (outptr + 4 > outend,\ - 0)) \ - { \ - result = __GCONV_FULL_OUTPUT; \ - break; \ - } \ - *outptr++ = ESC; \ - *outptr++ = '$'; \ - *outptr++ = '('; \ - *outptr++ = 'C'; \ - set = KSC5601_set; \ - \ - if (__builtin_expect (outptr + 2 > outend,\ - 0)) \ - { \ - result = __GCONV_FULL_OUTPUT; \ - break; \ - } \ - *outptr++ = buf[0]; \ - *outptr++ = buf[1]; \ - } \ - else \ - { \ - const struct gap *rp = from_idx; \ - unsigned char gch = 0; \ - \ - while (ch > rp->end) \ - ++rp; \ - if (ch >= rp->start) \ - { \ - ch = ch - 0xa0 + rp->idx; \ - gch = iso88597_from_ucs4[ch]; \ - } \ - \ - if (__builtin_expect (gch, 1) != 0) \ - { \ - /* We use ISO 8859-7 greek. */ \ - if (__builtin_expect (outptr + 3 \ - > outend, 0)) \ - { \ - result = __GCONV_FULL_OUTPUT; \ - break; \ - } \ - *outptr++ = ESC; \ - *outptr++ = '.'; \ - *outptr++ = 'F'; \ - set2 = ISO88597_set; \ - \ - if (__builtin_expect (outptr + 3 \ - > outend, 0)) \ - { \ - result = __GCONV_FULL_OUTPUT; \ - break; \ - } \ - *outptr++ = ESC; \ - *outptr++ = 'N'; \ - *outptr++ = gch; \ - } \ - else \ - { \ - STANDARD_ERR_HANDLER (4); \ - } \ - } \ + result = __GCONV_FULL_OUTPUT; \ + break; \ } \ + *outptr++ = ESC; \ + *outptr++ = 'N'; \ + *outptr++ = res; \ + result = __GCONV_OK; \ + break; \ } \ } \ } \ + \ + break; \ + \ + case japanese: \ + \ + /* Try JIS X 0201 Roman. */ \ + written = ucs4_to_jisx0201 (ch, buf); \ + if (written != __UNKNOWN_10646_CHAR \ + && buf[0] > 0x20 && buf[0] < 0x80) \ + { \ + if (set != JISX0201_Roman_set) \ + { \ + if (__builtin_expect (outptr + 3 > outend, 0)) \ + { \ + result = __GCONV_FULL_OUTPUT; \ + break; \ + } \ + *outptr++ = ESC; \ + *outptr++ = '('; \ + *outptr++ = 'J'; \ + set = JISX0201_Roman_set; \ + } \ + \ + if (__builtin_expect (outptr + 1 > outend, 0)) \ + { \ + result = __GCONV_FULL_OUTPUT; \ + break; \ + } \ + *outptr++ = buf[0]; \ + result = __GCONV_OK; \ + break; \ + } \ + \ + /* Try JIS X 0208. */ \ + written = ucs4_to_jisx0208 (ch, buf, 2); \ + if (written != __UNKNOWN_10646_CHAR) \ + { \ + if (set != JISX0208_1983_set) \ + { \ + if (__builtin_expect (outptr + 3 > outend, 0)) \ + { \ + result = __GCONV_FULL_OUTPUT; \ + break; \ + } \ + *outptr++ = ESC; \ + *outptr++ = '$'; \ + *outptr++ = 'B'; \ + set = JISX0208_1983_set; \ + } \ + \ + if (__builtin_expect (outptr + 2 > outend, 0)) \ + { \ + result = __GCONV_FULL_OUTPUT; \ + break; \ + } \ + *outptr++ = buf[0]; \ + *outptr++ = buf[1]; \ + result = __GCONV_OK; \ + break; \ + } \ + \ + if (__builtin_expect (var == iso2022jp, 0)) \ + /* Don't use the other Japanese character sets. */ \ + break; \ + \ + /* Try JIS X 0212. */ \ + written = ucs4_to_jisx0212 (ch, buf, 2); \ + if (written != __UNKNOWN_10646_CHAR) \ + { \ + if (set != JISX0212_set) \ + { \ + if (__builtin_expect (outptr + 4 > outend, 0)) \ + { \ + result = __GCONV_FULL_OUTPUT; \ + break; \ + } \ + *outptr++ = ESC; \ + *outptr++ = '$'; \ + *outptr++ = '('; \ + *outptr++ = 'D'; \ + set = JISX0212_set; \ + } \ + \ + if (__builtin_expect (outptr + 2 > outend, 0)) \ + { \ + result = __GCONV_FULL_OUTPUT; \ + break; \ + } \ + *outptr++ = buf[0]; \ + *outptr++ = buf[1]; \ + result = __GCONV_OK; \ + break; \ + } \ + \ + break; \ + \ + case chinese: \ + assert (var == iso2022jp2); \ + \ + /* Try GB 2312. */ \ + written = ucs4_to_gb2312 (ch, buf, 2); \ + if (written != __UNKNOWN_10646_CHAR) \ + { \ + if (set != GB2312_set) \ + { \ + if (__builtin_expect (outptr + 3 > outend, 0)) \ + { \ + result = __GCONV_FULL_OUTPUT; \ + break; \ + } \ + *outptr++ = ESC; \ + *outptr++ = '$'; \ + *outptr++ = 'A'; \ + set = GB2312_set; \ + } \ + \ + if (__builtin_expect (outptr + 2 > outend, 0)) \ + { \ + result = __GCONV_FULL_OUTPUT; \ + break; \ + } \ + *outptr++ = buf[0]; \ + *outptr++ = buf[1]; \ + result = __GCONV_OK; \ + break; \ + } \ + \ + break; \ + \ + case korean: \ + assert (var == iso2022jp2); \ + \ + /* Try KSC 5601. */ \ + written = ucs4_to_ksc5601 (ch, buf, 2); \ + if (written != __UNKNOWN_10646_CHAR) \ + { \ + if (set != KSC5601_set) \ + { \ + if (__builtin_expect (outptr + 4 > outend, 0)) \ + { \ + result = __GCONV_FULL_OUTPUT; \ + break; \ + } \ + *outptr++ = ESC; \ + *outptr++ = '$'; \ + *outptr++ = '('; \ + *outptr++ = 'C'; \ + set = KSC5601_set; \ + } \ + \ + if (__builtin_expect (outptr + 2 > outend, 0)) \ + { \ + result = __GCONV_FULL_OUTPUT; \ + break; \ + } \ + *outptr++ = buf[0]; \ + *outptr++ = buf[1]; \ + result = __GCONV_OK; \ + break; \ + } \ + \ + break; \ + \ + case other: \ + assert (var == iso2022jp2); \ + \ + /* Try JIS X 0201 Kana. This is not officially part \ + of ISO-2022-JP-2, according to RFC 1554. Therefore \ + we try this only after all other attempts. */ \ + written = ucs4_to_jisx0201 (ch, buf); \ + if (written != __UNKNOWN_10646_CHAR && buf[0] >= 0x80) \ + { \ + if (set != JISX0201_Kana_set) \ + { \ + if (__builtin_expect (outptr + 3 > outend, 0)) \ + { \ + result = __GCONV_FULL_OUTPUT; \ + break; \ + } \ + *outptr++ = ESC; \ + *outptr++ = '('; \ + *outptr++ = 'I'; \ + set = JISX0201_Kana_set; \ + } \ + \ + if (__builtin_expect (outptr + 1 > outend, 0)) \ + { \ + result = __GCONV_FULL_OUTPUT; \ + break; \ + } \ + *outptr++ = buf[0] - 0x80; \ + result = __GCONV_OK; \ + break; \ + } \ + \ + break; \ + \ + default: \ + abort (); \ + } \ + while (result == __GCONV_ILLEGAL_INPUT \ + && (conversion_list = CVLIST_REST (conversion_list)) != 0);\ + \ + if (result == __GCONV_FULL_OUTPUT) \ + break; \ + \ + if (result == __GCONV_ILLEGAL_INPUT) \ + { \ + STANDARD_ERR_HANDLER (4); \ } \ } \ } \ @@ -910,8 +1066,9 @@ gconv_end (struct __gconv_step *data) #define LOOP_NEED_FLAGS #define EXTRA_LOOP_DECLS , enum variant var, int *setp #define INIT_PARAMS int set = *setp & CURRENT_SEL_MASK; \ - int set2 = *setp & CURRENT_ASSIGN_MASK -#define UPDATE_PARAMS *setp = set | set2 + int set2 = *setp & CURRENT_ASSIGN_MASK; \ + int tag = *setp & CURRENT_TAG_MASK; +#define UPDATE_PARAMS *setp = set | set2 | tag #include <iconv/loop.c> diff --git a/iconvdata/iso-2022-kr.c b/iconvdata/iso-2022-kr.c index 6547768ab2..4033dc0632 100644 --- a/iconvdata/iso-2022-kr.c +++ b/iconvdata/iso-2022-kr.c @@ -1,5 +1,5 @@ /* Conversion module for ISO-2022-KR. - Copyright (C) 1998, 1999, 2000 Free Software Foundation, Inc. + Copyright (C) 1998, 1999, 2000, 2001 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Ulrich Drepper <drepper@cygnus.com>, 1998. @@ -221,7 +221,6 @@ enum #define BODY \ { \ uint32_t ch; \ - size_t written = 0; \ \ ch = get32 (inptr); \ \ @@ -241,16 +240,18 @@ enum } \ \ *outptr++ = ch; \ - written = 1; \ } \ else \ { \ char buf[2]; \ + size_t written; \ \ written = ucs4_to_ksc5601 (ch, buf, 2); \ \ if (__builtin_expect (written, 0) == __UNKNOWN_10646_CHAR) \ { \ + UNICODE_TAG_HANDLER (ch, 4); \ + \ /* Illegal character. */ \ STANDARD_ERR_HANDLER (4); \ } \ diff --git a/iconvdata/iso646.c b/iconvdata/iso646.c index 8de2639574..6668d9db7b 100644 --- a/iconvdata/iso646.c +++ b/iconvdata/iso646.c @@ -1,5 +1,5 @@ /* Conversion to and from the various ISO 646 CCS. - Copyright (C) 1998, 1999, 2000 Free Software Foundation, Inc. + Copyright (C) 1998, 1999, 2000, 2001 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Ulrich Drepper <drepper@cygnus.com>, 1998. @@ -879,7 +879,10 @@ gconv_end (struct __gconv_step *data) break; \ default: \ if (__builtin_expect (ch, 0) > 0x7f) \ - failure = __GCONV_ILLEGAL_INPUT; \ + { \ + UNICODE_TAG_HANDLER (ch, 4); \ + failure = __GCONV_ILLEGAL_INPUT; \ + } \ break; \ } \ \ diff --git a/iconvdata/iso8859-1.c b/iconvdata/iso8859-1.c index 9e5007f77d..7b2ed45c3a 100644 --- a/iconvdata/iso8859-1.c +++ b/iconvdata/iso8859-1.c @@ -1,5 +1,5 @@ /* Conversion to and from ISO 8859-1. - Copyright (C) 1997, 1998, 1999, 2000 Free Software Foundation, Inc. + Copyright (C) 1997, 1998, 1999, 2000, 2001 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Ulrich Drepper <drepper@cygnus.com>, 1997. @@ -48,6 +48,8 @@ uint32_t ch = *((const uint32_t *) inptr); \ if (__builtin_expect (ch, 0) > 0xff) \ { \ + UNICODE_TAG_HANDLER (ch, 4); \ + \ /* We have an illegal character. */ \ STANDARD_ERR_HANDLER (4); \ } \ diff --git a/iconvdata/iso_6937-2.c b/iconvdata/iso_6937-2.c index 5f8956e6b5..0c2d6d251b 100644 --- a/iconvdata/iso_6937-2.c +++ b/iconvdata/iso_6937-2.c @@ -1,5 +1,5 @@ /* Generic conversion to and from ISO 6937-2. - Copyright (C) 1998, 1999, 2000 Free Software Foundation, Inc. + Copyright (C) 1998, 1999, 2000, 2001 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Ulrich Drepper <drepper@cygnus.com>, 1998. @@ -563,6 +563,7 @@ static const char from_ucs4[][2] = cp = "\xd5"; \ break; \ default: \ + UNICODE_TAG_HANDLER (ch, 4); \ /* Illegal characters. */ \ cp = NULL; \ break; \ diff --git a/iconvdata/iso_6937.c b/iconvdata/iso_6937.c index 74b899005a..e99bfa9ed9 100644 --- a/iconvdata/iso_6937.c +++ b/iconvdata/iso_6937.c @@ -1,5 +1,5 @@ /* Generic conversion to and from ISO 6937. - Copyright (C) 1997, 1998, 1999, 2000 Free Software Foundation, Inc. + Copyright (C) 1997, 1998, 1999, 2000, 2001 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Ulrich Drepper <drepper@cygnus.com>, 1997. @@ -536,6 +536,7 @@ static const char from_ucs4[][2] = cp = "\xd5"; \ break; \ default: \ + UNICODE_TAG_HANDLER (ch, 4); \ cp = NULL; \ fail = 1; \ } \ diff --git a/iconvdata/johab.c b/iconvdata/johab.c index a631140f09..b92906cbaf 100644 --- a/iconvdata/johab.c +++ b/iconvdata/johab.c @@ -1,5 +1,5 @@ /* Mapping tables for JOHAB handling. - Copyright (C) 1998, 1999, 2000 Free Software Foundation, Inc. + Copyright (C) 1998, 1999, 2000, 2001 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Jungshik Shin <jshin@pantheon.yale.edu> and Ulrich Drepper <drepper@cygnus.com>, 1998. @@ -426,6 +426,7 @@ johab_sym_hanja_to_ucs (uint_fast32_t idx, uint_fast32_t c1, uint_fast32_t c2) if (__builtin_expect (written, 1) == __UNKNOWN_10646_CHAR \ || (outptr[0] == 0x22 && outptr[1] > 0x65)) \ { \ + UNICODE_TAG_HANDLER (ch, 4); \ STANDARD_ERR_HANDLER (4); \ } \ \ diff --git a/iconvdata/sjis.c b/iconvdata/sjis.c index 7bbfef3e5d..cb5014ccb3 100644 --- a/iconvdata/sjis.c +++ b/iconvdata/sjis.c @@ -1,5 +1,5 @@ /* Mapping tables for SJIS handling. - Copyright (C) 1997, 1998, 1999, 2000 Free Software Foundation, Inc. + Copyright (C) 1997, 1998, 1999, 2000, 2001 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Ulrich Drepper <drepper@cygnus.com>, 1997. @@ -4471,8 +4471,11 @@ static const char from_ucs4_extra[0x100][2] = && __builtin_expect (ch, 0xff01) <= 0xffef) \ cp = from_ucs4_extra[ch - 0xff00]; \ else \ - /* Illegal character. */ \ - cp = ""; \ + { \ + UNICODE_TAG_HANDLER (ch, 4); \ + /* Illegal character. */ \ + cp = ""; \ + } \ } \ else \ cp = from_ucs4_lat1[ch]; \ diff --git a/iconvdata/t.61.c b/iconvdata/t.61.c index 302226d8cc..4dab974012 100644 --- a/iconvdata/t.61.c +++ b/iconvdata/t.61.c @@ -1,5 +1,5 @@ /* Generic conversion to and from T.61. - Copyright (C) 1997, 1998, 1999, 2000 Free Software Foundation, Inc. + Copyright (C) 1997, 1998, 1999, 2000, 2001 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Ulrich Drepper <drepper@cygnus.com>, 1997. @@ -468,6 +468,8 @@ static const char from_ucs4[][2] = || __builtin_expect (ch, 0x2d8) > 0x2dd \ || __builtin_expect (ch, 0x2d8) == 0x02dc) \ { \ + UNICODE_TAG_HANDLER (ch, 4); \ + \ /* Illegal characters. */ \ STANDARD_ERR_HANDLER (4); \ } \ diff --git a/iconvdata/tst-table-charmap.sh b/iconvdata/tst-table-charmap.sh index 99c713c363..889595c8b7 100755 --- a/iconvdata/tst-table-charmap.sh +++ b/iconvdata/tst-table-charmap.sh @@ -1,5 +1,5 @@ #!/bin/sh -# Copyright (C) 2000 Free Software Foundation, Inc. +# Copyright (C) 2000, 2001 Free Software Foundation, Inc. # This file is part of the GNU C Library. # Contributed by Bruno Haible <haible@clisp.cons.org>, 2000. # @@ -30,6 +30,6 @@ case "$1" in ;; *) # New Unicode based format - sed -e 's,^%IRREVERSIBLE%,,' | grep '^<U....>[ ]*/x' | grep -v 'not a real character' | sed -e 's,<U\(....\)>[ ]*\([/x0-9A-Fa-f]*\).*$,\2 0x\1,' | tr abcdef ABCDEF | sed -e 's,/x\([0-9A-F][0-9A-F]\),\1,g' | sed -e 's,^,0x,' | sort | uniq | grep -v '^0x00 0x\([1-9A-F]...\|.[1-9A-F]..\|..[1-9A-F].\|...[1-9A-F]\)' + sed -e 's,^%IRREVERSIBLE%,,' | grep '^<U[0-9A-F][0-9A-F][0-9A-F][0-9A-F][0-9A-F]*>[ ]*/x' | grep -v 'not a real character' | sed -e 's,<U\(....\)>[ ]*\([/x0-9A-Fa-f]*\).*$,\2 0x\1,' -e 's,<U0*\([1-9A-F][0-9A-F][0-9A-F][0-9A-F][0-9A-F][0-9A-F]*\)>[ ]*\([/x0-9A-Fa-f]*\).*$,\2 0x\1,' | tr abcdef ABCDEF | sed -e 's,/x\([0-9A-F][0-9A-F]\),\1,g' | sed -e 's,^,0x,' | sort | uniq | grep -v '^0x00 0x\([1-9A-F]...\|.[1-9A-F]..\|..[1-9A-F].\|...[1-9A-F]\)' ;; esac diff --git a/iconvdata/tst-table-from.c b/iconvdata/tst-table-from.c index b24632e1b9..067c600892 100644 --- a/iconvdata/tst-table-from.c +++ b/iconvdata/tst-table-from.c @@ -28,6 +28,9 @@ #include <iconv.h> #include <errno.h> +/* If nonzero, ignore conversions outside Unicode plane 0. */ +static int bmp_only; + /* Converts a byte buffer to a hexadecimal string. */ static const char* hexbuf (unsigned char buf[], unsigned int buflen) @@ -162,6 +165,10 @@ utf8_decode (const unsigned char *out, unsigned int outlen) out += 1; outlen -= 1; } + if (bmp_only && strlen (p) > 6) + /* Ignore conversions outside Unicode plane 0. */ + return NULL; + p += strlen (p); } @@ -173,6 +180,7 @@ main (int argc, char *argv[]) { const char *charset; iconv_t cd; + int search_depth; if (argc != 2) { @@ -188,6 +196,12 @@ main (int argc, char *argv[]) exit (1); } + /* When testing UTF-8 or GB18030, stop at 0x10000, otherwise the output + file gets too big. */ + bmp_only = (strcmp (charset, "UTF-8") == 0 + || strcmp (charset, "GB18030") == 0); + search_depth = (strcmp (charset, "UTF-8") == 0 ? 3 : 4); + { unsigned char out[6]; unsigned char buf[4]; @@ -203,8 +217,9 @@ main (int argc, char *argv[]) } else if (result > 0) { - printf ("0x%02X\t%s\n", - i0, utf8_decode (out, result)); + const char *unicode = utf8_decode (out, result); + if (unicode != NULL) + printf ("0x%02X\t%s\n", i0, unicode); } else { @@ -217,8 +232,9 @@ main (int argc, char *argv[]) } else if (result > 0) { - printf ("0x%02X%02X\t%s\n", - i0, i1, utf8_decode (out, result)); + const char *unicode = utf8_decode (out, result); + if (unicode != NULL) + printf ("0x%02X%02X\t%s\n", i0, i1, unicode); } else { @@ -231,10 +247,12 @@ main (int argc, char *argv[]) } else if (result > 0) { - printf ("0x%02X%02X%02X\t%s\n", - i0, i1, i2, utf8_decode (out, result)); + const char *unicode = utf8_decode (out, result); + if (unicode != NULL) + printf ("0x%02X%02X%02X\t%s\n", + i0, i1, i2, unicode); } - else if (strcmp (charset, "UTF-8")) + else if (search_depth > 3) { for (i3 = 0; i3 < 0x100; i3++) { @@ -245,9 +263,11 @@ main (int argc, char *argv[]) } else if (result > 0) { - printf ("0x%02X%02X%02X%02X\t%s\n", - i0, i1, i2, i3, - utf8_decode (out, result)); + const char *unicode = + utf8_decode (out, result); + if (unicode != NULL) + printf ("0x%02X%02X%02X%02X\t%s\n", + i0, i1, i2, i3, unicode); } else { diff --git a/iconvdata/tst-table-to.c b/iconvdata/tst-table-to.c index c763c845c9..0a6ba8a8da 100644 --- a/iconvdata/tst-table-to.c +++ b/iconvdata/tst-table-to.c @@ -1,4 +1,4 @@ -/* Copyright (C) 2000 Free Software Foundation, Inc. +/* Copyright (C) 2000-2001 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Bruno Haible <haible@clisp.cons.org>, 2000. @@ -32,6 +32,7 @@ main (int argc, char *argv[]) { const char *charset; iconv_t cd; + int bmp_only; if (argc != 2) { @@ -40,22 +41,38 @@ main (int argc, char *argv[]) } charset = argv[1]; - cd = iconv_open (charset, "UCS-2"); + cd = iconv_open (charset, "UTF-8"); if (cd == (iconv_t)(-1)) { perror ("iconv_open"); return 1; } + /* When testing UTF-8 or GB18030, stop at 0x10000, otherwise the output + file gets too big. */ + bmp_only = (strcmp (charset, "UTF-8") == 0 + || strcmp (charset, "GB18030") == 0); + { unsigned int i; unsigned char buf[10]; - for (i = 0; i < 0x10000; i++) + for (i = 0; i < (bmp_only ? 0x10000 : 0x30000); i++) { - unsigned short in = i; - const char *inbuf = (const char *) ∈ - size_t inbytesleft = sizeof (unsigned short); + unsigned char in[6]; + unsigned int incount = + (i < 0x80 ? (in[0] = i, 1) + : i < 0x800 ? (in[0] = 0xc0 | (i >> 6), + in[1] = 0x80 | (i & 0x3f), 2) + : i < 0x10000 ? (in[0] = 0xe0 | (i >> 12), + in[1] = 0x80 | ((i >> 6) & 0x3f), + in[2] = 0x80 | (i & 0x3f), 3) + : /* i < 0x200000 */ (in[0] = 0xf0 | (i >> 18), + in[1] = 0x80 | ((i >> 12) & 0x3f), + in[2] = 0x80 | ((i >> 6) & 0x3f), + in[3] = 0x80 | (i & 0x3f), 4)); + const char *inbuf = (const char *) in; + size_t inbytesleft = incount; char *outbuf = (char *) buf; size_t outbytesleft = sizeof (buf); size_t result = iconv (cd, @@ -78,7 +95,7 @@ main (int argc, char *argv[]) if (inbytesleft != 0 || outbytesleft == sizeof (buf)) { fprintf (stderr, "0x%02X: inbytes = %ld, outbytes = %ld\n", i, - (long) (sizeof (unsigned short) - inbytesleft), + (long) (incount - inbytesleft), (long) (sizeof (buf) - outbytesleft)); return 1; } diff --git a/iconvdata/tst-table.sh b/iconvdata/tst-table.sh index e38b88ee37..f00d3eef3e 100755 --- a/iconvdata/tst-table.sh +++ b/iconvdata/tst-table.sh @@ -1,5 +1,5 @@ #!/bin/sh -# Copyright (C) 2000 Free Software Foundation, Inc. +# Copyright (C) 2000, 2001 Free Software Foundation, Inc. # This file is part of the GNU C Library. # Contributed by Bruno Haible <haible@clisp.cons.org>, 2000. # @@ -61,8 +61,15 @@ diff ${objpfx}tst-${charset}.table ${objpfx}tst-${charset}.inverse.table | \ grep '^[<>]' | sed -e 's,^. ,,' > ${objpfx}tst-${charset}.irreversible.table # Check 1: charmap and iconv forward should be identical. -cmp -s ${objpfx}tst-${charset}.charmap.table ${objpfx}tst-${charset}.table || -exit 1 +if test ${charset} = GB18030; then + grep '0x....$' < ${objpfx}tst-${charset}.charmap.table \ + > ${objpfx}tst-${charset}.truncated.table + cmp -s ${objpfx}tst-${charset}.truncated.table ${objpfx}tst-${charset}.table || + exit 1 +else + cmp -s ${objpfx}tst-${charset}.charmap.table ${objpfx}tst-${charset}.table || + exit 1 +fi # Check 2: the difference between the two iconv directions. if test -f ${irreversible}; then diff --git a/iconvdata/uhc.c b/iconvdata/uhc.c index c191b1127e..c3ff69937e 100644 --- a/iconvdata/uhc.c +++ b/iconvdata/uhc.c @@ -1,5 +1,5 @@ /* Mapping tables for UHC handling. - Copyright (C) 1998, 1999, 2000 Free Software Foundation, Inc. + Copyright (C) 1998, 1999, 2000, 2001 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Jungshik Shin <jshin@pantheon.yale.edu>, 1998. @@ -3243,6 +3243,7 @@ static const char uhc_hangul_from_ucs[11172][2] = } \ if (__builtin_expect (written, 0) == __UNKNOWN_10646_CHAR) \ { \ + UNICODE_TAG_HANDLER (ch, 4); \ STANDARD_ERR_HANDLER (4); \ } \ \ diff --git a/iconvdata/unicode.c b/iconvdata/unicode.c index 89ec81f90b..f2f46c6c77 100644 --- a/iconvdata/unicode.c +++ b/iconvdata/unicode.c @@ -1,5 +1,5 @@ /* Conversion module for Unicode - Copyright (C) 1999, 2000 Free Software Foundation, Inc. + Copyright (C) 1999, 2000, 2001 Free Software Foundation, Inc. This file is part of the GNU C Library. Contributed by Ulrich Drepper <drepper@cygnus.com>, 1999. @@ -154,6 +154,7 @@ gconv_end (struct __gconv_step *data) \ if (__builtin_expect (c, 0) >= 0x10000) \ { \ + UNICODE_TAG_HANDLER (c, 4); \ STANDARD_ERR_HANDLER (4); \ } \ else if (__builtin_expect (c >= 0xd800 && c < 0xe000, 0)) \ |