about summary refs log tree commit diff
path: root/src/locale/iconv.c
Commit message (Collapse)AuthorAgeFilesLines
* fix output size handling for multi-unicode-char big5-hkscs charactersRich Felker2018-06-011-5/+13
| | | | | | | | | | | | | | | | | | | | since this iconv implementation's output is stateless, it's necessary to know before writing anything to the output buffer whether the conversion of the current input character will fit. previously we used a hard-coded table of the output size needed for each supported output encoding, but failed to update the table when adding support for conversion to jis-based encodings and again when adding separate encoding identifiers for implicit-endianness utf-16/32 and ucs-2/4 variants, resulting in out-of-bound table reads and incorrect size checks. no buffer overflow was possible, but the affected characters could be converted incorrectly, and iconv could potentially produce an incorrect return value as a result. remove the hard-coded table, and instead perform the recursive iconv conversion to a temporary buffer, measuring the output size and transferring it to the actual output buffer only if the whole converted result fits.
* fix iconv mapping of big5-hkscs characters that map to two unicode charsRich Felker2018-06-011-1/+1
| | | | | | | | | | | | | this case is handled with a recursive call to iconv using a specially-constructed conversion descriptor. the constant 0 was used as the offset for utf-8, since utf-8 appears first in the charmaps table, but the offset used needs to point into the charmap entry, past the name/aliases at the beginning, to the byte identifying the encoding. as a result of this error, junk was produced. instead, call find_charmap so we don't have to hard-code a nontrivial offset. with this change, the code has been tested and found to work in the case of converting the affected hkscs characters to utf-8.
* fix iconv conversion to UTF-32 with implicit (big) endiannessWill Dietz2018-05-091-0/+2
| | | | | | | | | | | maintainer's notes: commit 95c6044e2ae85846330814c4ac5ebf4102dbe02c split UTF-32 and UTF-32BE but neglected to add a case for the former as a destination encoding, resulting in it wrongly being handled by the default case. the intent was that the value of the macro be chosen to encode "big endian" in the low bits, so that no code would be needed, but this was botched; instead, handle it the way UCS2 is handled.
* fix iconv buffer overflow converting to legacy JIS-based encodingsWill Dietz2018-05-091-0/+1
| | | | | | | | | | maintainer's notes: commit a223dbd27ae36fe53f9f67f86caf685b729593fc added the reverse conversions to JIS-based encodings, but omitted the check for remining buffer space in the case where the next character to be written was single-byte, allowing conversion to continue past the end of the destination buffer.
* fix iconv output of surrogate pairs in ucs2Rich Felker2017-12-181-1/+1
| | | | | | in the unified code for handling utf-16 and ucs2 output, the check for ucs2 wrongly looked at the source charset rather than the destination charset.
* add support for BOM-determined-endian UCS2, UTF-16, and UTF-32 to iconvRich Felker2017-12-181-3/+40
| | | | | | | | | | | | previously, the charset names without endianness specified were always interpreted as big endian. unicode specifies that UTF-16 and UTF-32 have BOM-determined endianness if BOM is present, and are otherwise big endian. since commit 5b546faa67544af395d6407553762b37e9711157 added support for stateful encodings, it is now possible to implement BOM support via the conversion descriptor state. for conversions to these charsets, the output is always big endian and does not have a BOM.
* add reverse iconv mappings for JIS-based encodingsRich Felker2017-11-141-1/+97
| | | | | | | these encodings are still commonly used in messaging protocols and such. the reverse mapping is implemented as a binary search of a list of the jis 0208 characters in unicode order; the existing forward table is used to perform the comparison in the search.
* generalize iconv framework for 8-bit codepagesRich Felker2017-11-131-11/+16
| | | | | | | | | | | | | | | | | | | previously, 8-bit codepages could only remap the high 128 bytes; the low range was assumed/forced to agree with ascii. interpretation of codepage table headers has been changed so that it's possible to represent mappings for up to 256 slots (fewer if the initial portion of the map is elided because it coincides with unicode codepoints). this requires consuming a bit more of the 10-bit space of characters that can be represented in 8-bit codepages, but there's still a plenty left. the size of the legacy_chars table is actually reduced now by eliding the first 256 entries and considering them to map implicitly via the identity map. before these changes, there seem to have been minor bugs/omissions in codepage table generation, so it's likely that some actual bug fixes are silently included in this commit. round-trip testing of a few codepages was performed on the new version of the code, but no differential testing against the old version was done.
* add iso-2022-jp support (decoding only) to iconvRich Felker2017-11-101-2/+45
| | | | | | | | | | this implementation aims to match the baseline defined by rfc1468 (the original mime charset definition) plus the halfwidth katakana extension included in the whatwg definition of the charset. rejection of si/so controls and newlines in doublebyte state are not currently enforced. the jis x 0201 mode is currently interpreted as having the yen sign and overline character in place of backslash and tilde; ascii mode has the standard ascii characters in those slots.
* add iconv framework for decoding stateful encodingsRich Felker2017-11-101-3/+22
| | | | | | | assuming pointers obtained from malloc have some nonzero alignment, repurpose the low bit of iconv_t as an indicator that the descriptor is a stateless value representing the source and destination character encodings.
* simplify/optimize iconv utf-8 caseRich Felker2017-11-101-4/+3
| | | | | | the special case where mbrtowc returns 0 but consumed 1 byte of input does not need to be considered, because the short-circuit for low bytes already covered that case.
* handle ascii range individually in each iconv caseRich Felker2017-11-101-2/+10
| | | | | | | short-circuiting low bytes before the switch precluded support for character encodings that don't coincide with ascii in this range. this limitation affected iso-2022 encodings, which use the esc byte to introduce a shift sequence, and things like ebcdic.
* move iconv_close to its own translation unitRich Felker2017-11-101-5/+0
| | | | | | | this is in preparation to support stateful conversion descriptors, which are necessarily allocated and thus must be freed in iconv_close. putting it in a separate TU will avoid pulling in free if iconv_close is not referenced.
* refactor iconv conversion descriptor encoding/decodingRich Felker2017-11-101-6/+20
| | | | | | | | | | this change is made to avoid having assumptions about the encoding spread out across the file, and to facilitate future change to a form that can accommodate allocted, stateful descriptors when needed. this commit should not produce any functional changes; with the compiler tested the only change to code generation was minor reordering of local variables on stack.
* fix iconv conversions for iso88592-iso885916Bartosz Brachaczek2017-06-201-1/+1
| | | | | | | commit 97bd6b09dbe7478d5a90a06ecd9e5b59389d8eb9 refactored the table lookup into a function and introduced an error in index computation. the error caused garbage to be read from the table if the given charmap had a non-zero number of elided entries.
* fix iconv conversions to legacy 8bit encodingsRich Felker2017-05-271-9/+12
| | | | | | | | | | there was missing reverse-conversion logic for the case, handled specially in the character set tables, where a byte represents a unicode codepoint with the same value. this patch adds code to handle the case, and refactors the two-level 10-bit table lookup for legacy character sets into a function to avoid repeating it yet another time as part of the fix.
* byte-based C locale, phase 2: stdio and iconv (multibyte callers)Rich Felker2015-06-161-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | this patch adjusts libc components which use the multibyte functions internally, and which depend on them operating in a particular encoding, to make the appropriate locale changes before calling them and restore the calling thread's locale afterwards. activating the byte-based C locale without these changes would cause regressions in stdio and iconv. in the case of iconv, the current implementation was simply using the multibyte functions as UTF-8 conversions. setting a multibyte UTF-8 locale for the duration of the iconv operation allows the code to continue working. in the case of stdio, POSIX requires that FILE streams have an encoding rule bound at the time of setting wide orientation. as long as all locales, including the C locale, used the same encoding, treating high bytes as UTF-8, there was no need to store an encoding rule as part of the stream's state. a new locale field in the FILE structure points to the locale that should be made active during fgetwc/fputwc/ungetwc on the stream. it cannot point to the locale active at the time the stream becomes oriented, because this locale could be mutable (the global locale) or could be destroyed (locale_t objects produced by newlocale) before the stream is closed. instead, a pointer to the static C or C.UTF-8 locale object added in commit commit aeeac9ca5490d7d90fe061ab72da446c01ddf746 is used. this is valid since categories other than LC_CTYPE will not affect these functions.
* remove outdated and misleading comment in iconv.cRich Felker2015-05-211-6/+0
| | | | | the comment claimed that EUC/GBK/Big5 are not implemented, which has been incorrect since commit 19b4a0a20efc6b9df98b6a43536ecdd628ba4643.
* in iconv_open, accept "CHAR" and "" as aliases for "UTF-8"Rich Felker2015-05-211-1/+2
| | | | | | | | while not a requirement, it's common convention in other iconv implementations to accept "CHAR" as an alias for nl_langinfo(CODESET), meaning the encoding used for char[] strings in the current locale, and also "" as an alternate form. supporting this is not costly and improves compatibility.
* add hkscs/big5-2003/eten extensions to iconv big5Rich Felker2013-08-171-4/+33
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | with these changes, the character set implemented as "big5" in musl is a pure superset of cp950, the canonical "big5", and agrees with the normative parts of Unicode. this means it has minor differences from both hkscs and big5-2003: - the range A2CC-A2CE maps to CJK ideographs rather than numerals, contrary to changes made in big5-2003. - C6CD maps to a CJK ideograph rather than its corresponding Kangxi radical character, contrary to changes made in hkscs. - F9FE maps to U+2593 rather than U+FFED. of these differences, none but the last are visually distinct, and the last is a character used purely for text-based graphics, not to convey linguistic content. should there be future demand for strict conformance to big5-2003 or hkscs mappings, the present charset aliases can be replaced with distinct variants. reportedly there are other non-standard big5 extensions in common use in Taiwan and perhaps elsewhere, which could also be added as layers on top of the existing big5 support. there may be additional characters which should be added to the hkscs table: the whatwg standard for big5 defines what appears to be a superset of hkscs.
* add Big5 charset support to iconvRich Felker2013-08-071-0/+18
| | | | | | at this point, it is just the common base charset equivalent to Windows CP 950, with no further extensions. HKSCS and possibly other supersets will be added later. other aliases may need to be added too.
* iconv support for legacy Korean encodingsRich Felker2013-08-051-0/+38
| | | | | | | | | | | | | | | | | | | | | like for other character sets, stateful iso-2022 form is not supported yet but everything else should work. all charset aliases are treated the same, as Windows codepage 949, because reportedly the EUC-KR charset name is in widespread (mis?)usage in email and on the web for data which actually uses the extended characters outside the standard 93x94 grid. this could easily be changed if desired. the principle of this converter for handling the giant bulk of rare Hangul syllables outside of the standard KS X 1001 93x94 grid is the same as the GB18030 converter's treatment of non-explicitly-coded Unicode codepoints: sequences in the extension range are mapped to an integer index N, and the converter explicitly computes the Nth Hangul syllable not explicitly encoded in the character map. empirically, this requires at most 7 passes over the grid. this approach reduces the table size required for Korean legacy encodings from roughly 44k to 17k and should have minimal performance impact on real-world text conversions since the "slow" characters are rare. where it does have impact, the cost is merely a large constant time factor.
* fix iconv conversion to legacy 8bit codepagesRich Felker2013-06-261-2/+2
| | | | | this seems to have been a simple copy-and-paste error from the code for converting from legacy codepages.
* use restrict everywhere it's required by c99 and/or posix 2008Rich Felker2012-09-061-1/+1
| | | | | | | | to deal with the fact that the public headers may be used with pre-c99 compilers, __restrict is used in place of restrict, and defined appropriately for any supported compiler. we also avoid the form [restrict] since older versions of gcc rejected it due to a bug in the original c99 standard, and instead use the form *restrict.
* fix multiple iconv bugs reading utf-16/32 and wchar_tRich Felker2012-06-181-8/+8
|
* fix iconv dest utf-16: unavailable chars must be replaced; EILSEQ is wrongRich Felker2012-06-181-2/+2
|
* fix erroneous utf-16 encoding with surrogates in iconvRich Felker2012-06-181-0/+1
| | | | apparently this was never tested before.
* fix major breakage in iconv, bogus rejecting of dest charsetsRich Felker2012-04-211-1/+1
|
* gb18030 support in iconv (only from, not to)Rich Felker2011-07-121-2/+51
| | | | | also support (and restrict to subsets) older chinese sets, and explicitly refuse to convert to cjk (since there's no code for it yet)
* legacy japanese charset support in iconv (only from, not to)Rich Felker2011-07-121-0/+47
|
* simplify iconv and support more legacy codepagesRich Felker2011-07-121-352/+54
|
* iconv was not returning -1 on most failureRich Felker2011-07-031-0/+2
| | | | | this broke most uses of iconv in real-world programs, especially glib's iconv wrappers.
* fix breakage due to converting a return type to size_t in iconv...Rich Felker2011-04-071-1/+1
|
* fix all implicit conversion between signed/unsigned pointersRich Felker2011-03-251-11/+11
| | | | | | | sadly the C language does not specify any such implicit conversion, so this is not a matter of just fixing warnings (as gcc treats it) but actual errors. i would like to revisit a number of these changes and possibly revise the types used to reduce the number of casts required.
* use a more-correct integer type, and silence 64-bit warnings as a bonusRich Felker2011-02-131-2/+2
|
* initial check-in, version 0.5.0 v0.5.0Rich Felker2011-02-121-0/+568