Shin Kojima <shin@xxxxxxxxxx> writes:
> Offset positions should not be counted by byte length, but by actual
> character length.
> ...
> # escape tabs (convert tabs to spaces)
> sub untabify {
> - my $line = shift;
> + my $line = to_utf8(shift);
>
> while ((my $pos = index($line, "\t")) != -1) {
> if (my $count = (8 - ($pos % 8))) {
Some codepaths in the resulting codeflow look even hackier than they
already are. For example, format_rem_add_lines_pair() calls
untabify() and then feeds its result to esc_html(). The first thing
done in esc_html() is to call to_utf8(). I know that to_utf8()
cheats and leaves the incoming string as-is if it is already UTF-8,
so this may be a safe conversion, but ideally we should be able to
say "function X takes non-UTF8 and works on it", "function Y takes
UTF8 and works on it", and "function Z takes non-UTF8 and gives UTF8
data back" for each functions clearly, not "function W can take
either UTF8 or any other garbage and tries to return UTF8".
Also, does it even "fix" the problem to use to_utf8() here in the
first place? Untabify is about aligning the character after a HT to
multiple of 8 display position, so we'd want measure display width,
which is not the same as either byte count or char count.
