Encode-Locale-1.03/000755 000765 000024 00000000000 11715507436 014306 5ustar00gislestaff000000 000000 Encode-Locale-1.03/Changes000644 000765 000024 00000004440 11715507233 015576 0ustar00gislestaff000000 000000 ## 2012-01-11 Release 1.03 Documentation spelling fixes and tweaks to improve testing on Windows. ## 2011-03-28 Release 1.02 Added supporting hacks for GB18030 and roman8. [RT#66373] [RT#67010] ## 2011-02-22 Release 1.01 Avoid compilation warnings (RT#65975). Thanks to Goro Fuji. ## 2011-01-23 Release 1.00 Documentation tweaks. ## 2010-10-23 Release 0.04 Look up the ANSI code page on Windows and alias this one as the "locale" encoding. Providide the reinit() function to be called if something is changed in the locale and you need to re-initialize the encodings set up by this module. Improved documentation. ## 2010-10-20 Release 0.03 Changed the default for decode_argv() to match Encode's default. It became too strange to tell people to pass FB_DEFAULT to get the non-default behaviour. Changed $ENCODING_FS into $ENCODING_LOCALE_FS (as already documented below), but not implemented as such. Workaround for test failure where the Encode does not know about the "646" encoding alias. Documentation tweaks. ## 2010-10-13 Release 0.02 ...where I realized that I could not get away with a single locale encoding. Now `Encode::Locale` provides 4 encoding names that often will map to the same underlying encoding. I've used the following names: locale $ENCODING_LOCALE locale_fs $ENCODING_LOCALE_FS console_in $ENCODING_CONSOLE_IN console_out $ENCODING_CONSOLE_OUT The first one is the encoding specified by the POSIX locale (or the equivalent on Windows). This can be set by the user. The second one (`locale_fs`) is the encoding that should be used when interfacing with the file system, that is the encoding of file names. For some systems (like Mac OS X) this is fixed system wide and the same for all users. Last; some systems allow the input and output encoding for data aimed at the console to differ so there are separate entries for these. For classic POSIX systems all 4 of these will all denote the same encoding. This release also introduce the function env() as a Unicode interface to the %ENV hash (the process environment variables). We don't want to decode the ENV %values in-place because this also affects what the child processes observes. The %ENV hash should always contain byte strings. ## 2010-10-11 Release 0.01 Initial release Encode-Locale-1.03/lib/000755 000765 000024 00000000000 11715507436 015054 5ustar00gislestaff000000 000000 Encode-Locale-1.03/Makefile.PL000644 000765 000024 00000002316 11715477251 016263 0ustar00gislestaff000000 000000 require 5.008; use ExtUtils::MakeMaker; WriteMakefile( NAME => 'Encode::Locale', VERSION_FROM => 'lib/Encode/Locale.pm', ABSTRACT_FROM => "lib/Encode/Locale.pm", AUTHOR => 'Gisle Aas ', LICENSE => "perl", MIN_PERL_VERSION => 5.008, PREREQ_PM => { 'Encode' => 2, 'Encode::Alias' => 0, }, META_MERGE => { resources => { repository => 'http://github.com/gisle/encode-locale', }, recommends => { 'I18N::Langinfo' => 0, 'Win32::Console' => 0, }, }, BUILD_REQUIRES => { Test => 0, }, ); BEGIN { # compatibility with older versions of MakeMaker my $developer = -d ".git"; my %mm_req = ( LICENCE => 6.31, META_MERGE => 6.45, META_ADD => 6.45, MIN_PERL_VERSION => 6.48, BUILD_REQUIRES => 6.56, ); undef(*WriteMakefile); *WriteMakefile = sub { my %arg = @_; for (keys %mm_req) { unless (eval { ExtUtils::MakeMaker->VERSION($mm_req{$_}) }) { warn "$_ $@" if $developer; delete $arg{$_}; } } ExtUtils::MakeMaker::WriteMakefile(%arg); }; } Encode-Locale-1.03/MANIFEST000644 000765 000024 00000000255 11715507436 015441 0ustar00gislestaff000000 000000 Changes lib/Encode/Locale.pm Makefile.PL MANIFEST This list of files README t/arg.t t/env.t META.yml Module meta-data (added by MakeMaker) Encode-Locale-1.03/META.yml000644 000765 000024 00000001306 11715507436 015557 0ustar00gislestaff000000 000000 --- #YAML:1.0 name: Encode-Locale version: 1.03 abstract: Determine the locale encoding author: - Gisle Aas license: perl distribution_type: module configure_requires: ExtUtils::MakeMaker: 0 build_requires: Test: 0 requires: Encode: 2 Encode::Alias: 0 perl: 5.008 resources: repository: http://github.com/gisle/encode-locale no_index: directory: - t - inc generated_by: ExtUtils::MakeMaker version 6.57_05 meta-spec: url: http://module-build.sourceforge.net/META-spec-v1.4.html version: 1.4 recommends: I18N::Langinfo: 0 Win32::Console: 0 Encode-Locale-1.03/README000644 000765 000024 00000000547 11715477251 015175 0ustar00gislestaff000000 000000 Encode-Locale ============== The purpose of this Perl module is try determine what encodings should be used when interfacing to various external interfaces. You will need perl-5.8 or better to use this module. © 2010 Gisle Aas ``. This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself. Encode-Locale-1.03/t/000755 000765 000024 00000000000 11715507436 014551 5ustar00gislestaff000000 000000 Encode-Locale-1.03/t/arg.t000644 000765 000024 00000000710 11715477251 015506 0ustar00gislestaff000000 000000 #!perl -w use strict; use Test; plan tests => 1; use Encode::Locale qw($ENCODING_LOCALE decode_argv); print "# ENCODING_LOCALE is $ENCODING_LOCALE\n"; decode_argv(); my $i; for my $arg (@ARGV) { print "# ", ++$i, ": \""; for (split(//, $arg)) { if (ord() > 32 && ord() < 128) { print $_; } elsif (ord() < 256) { printf "\\x%02X", ord(); } else { printf "\\x{%04X}", ord(); } } print "\"\n"; } # fake it :-) ok(1); Encode-Locale-1.03/t/env.t000644 000765 000024 00000001016 11715506257 015524 0ustar00gislestaff000000 000000 #!perl -w use strict; use Test; plan tests => 13; use Encode::Locale qw(env); $ENV{foo} = "bar"; ok(env("foo"), "bar"); ok(env("foo", "baz"), "bar"); ok(env("foo"), "baz"); ok($ENV{foo}, "baz"); ok(env("foo", undef), "baz"); ok(env("foo"), undef); ok(!exists $ENV{foo}); Encode::Locale::reinit("cp1252"); $ENV{"m\xf6ney"} = "\x80uro"; ok(env("m\xf6ney", "\x{20AC}"), "\x{20AC}uro"); ok(env("m\xf6ney"), "\x{20AC}"); ok($ENV{"m\xf6ney"}, "\x80"); ok(env("\x{20AC}", 1), undef); ok(env("\x{20AC}"), 1); ok($ENV{"\x80"}, 1); Encode-Locale-1.03/lib/Encode/000755 000765 000024 00000000000 11715507436 016251 5ustar00gislestaff000000 000000 Encode-Locale-1.03/lib/Encode/Locale.pm000644 000765 000024 00000025520 11715507102 020000 0ustar00gislestaff000000 000000 package Encode::Locale; use strict; our $VERSION = "1.03"; use base 'Exporter'; our @EXPORT_OK = qw( decode_argv env $ENCODING_LOCALE $ENCODING_LOCALE_FS $ENCODING_CONSOLE_IN $ENCODING_CONSOLE_OUT ); use Encode (); use Encode::Alias (); our $ENCODING_LOCALE; our $ENCODING_LOCALE_FS; our $ENCODING_CONSOLE_IN; our $ENCODING_CONSOLE_OUT; sub DEBUG () { 0 } sub _init { if ($^O eq "MSWin32") { unless ($ENCODING_LOCALE) { # Try to obtain what the Windows ANSI code page is eval { unless (defined &GetACP) { require Win32::API; Win32::API->Import('kernel32', 'int GetACP()'); }; if (defined &GetACP) { my $cp = GetACP(); $ENCODING_LOCALE = "cp$cp" if $cp; } }; } unless ($ENCODING_CONSOLE_IN) { # If we have the Win32::Console module installed we can ask # it for the code set to use eval { require Win32::Console; my $cp = Win32::Console::InputCP(); $ENCODING_CONSOLE_IN = "cp$cp" if $cp; $cp = Win32::Console::OutputCP(); $ENCODING_CONSOLE_OUT = "cp$cp" if $cp; }; # Invoking the 'chcp' program might also work if (!$ENCODING_CONSOLE_IN && (qx(chcp) || '') =~ /^Active code page: (\d+)/) { $ENCODING_CONSOLE_IN = "cp$1"; } } } unless ($ENCODING_LOCALE) { eval { require I18N::Langinfo; $ENCODING_LOCALE = I18N::Langinfo::langinfo(I18N::Langinfo::CODESET()); # Workaround of Encode < v2.25. The "646" encoding alias was # introduced in Encode-2.25, but we don't want to require that version # quite yet. Should avoid the CPAN testers failure reported from # openbsd-4.7/perl-5.10.0 combo. $ENCODING_LOCALE = "ascii" if $ENCODING_LOCALE eq "646"; # https://rt.cpan.org/Ticket/Display.html?id=66373 $ENCODING_LOCALE = "hp-roman8" if $^O eq "hpux" && $ENCODING_LOCALE eq "roman8"; }; $ENCODING_LOCALE ||= $ENCODING_CONSOLE_IN; } if ($^O eq "darwin") { $ENCODING_LOCALE_FS ||= "UTF-8"; } # final fallback $ENCODING_LOCALE ||= $^O eq "MSWin32" ? "cp1252" : "UTF-8"; $ENCODING_LOCALE_FS ||= $ENCODING_LOCALE; $ENCODING_CONSOLE_IN ||= $ENCODING_LOCALE; $ENCODING_CONSOLE_OUT ||= $ENCODING_CONSOLE_IN; unless (Encode::find_encoding($ENCODING_LOCALE)) { my $foundit; if (lc($ENCODING_LOCALE) eq "gb18030") { eval { require Encode::HanExtra; }; if ($@) { die "Need Encode::HanExtra to be installed to support locale codeset ($ENCODING_LOCALE), stopped"; } $foundit++ if Encode::find_encoding($ENCODING_LOCALE); } die "The locale codeset ($ENCODING_LOCALE) isn't one that perl can decode, stopped" unless $foundit; } # use Data::Dump; ddx $ENCODING_LOCALE, $ENCODING_LOCALE_FS, $ENCODING_CONSOLE_IN, $ENCODING_CONSOLE_OUT; } _init(); Encode::Alias::define_alias(sub { no strict 'refs'; no warnings 'once'; return ${"ENCODING_" . uc(shift)}; }, "locale"); sub _flush_aliases { no strict 'refs'; for my $a (keys %Encode::Alias::Alias) { if (defined ${"ENCODING_" . uc($a)}) { delete $Encode::Alias::Alias{$a}; warn "Flushed alias cache for $a" if DEBUG; } } } sub reinit { $ENCODING_LOCALE = shift; $ENCODING_LOCALE_FS = shift; $ENCODING_CONSOLE_IN = $ENCODING_LOCALE; $ENCODING_CONSOLE_OUT = $ENCODING_LOCALE; _init(); _flush_aliases(); } sub decode_argv { die if defined wantarray; for (@ARGV) { $_ = Encode::decode(locale => $_, @_); } } sub env { my $k = Encode::encode(locale => shift); my $old = $ENV{$k}; if (@_) { my $v = shift; if (defined $v) { $ENV{$k} = Encode::encode(locale => $v); } else { delete $ENV{$k}; } } return Encode::decode(locale => $old) if defined wantarray; } 1; __END__ =head1 NAME Encode::Locale - Determine the locale encoding =head1 SYNOPSIS use Encode::Locale; use Encode; $string = decode(locale => $bytes); $bytes = encode(locale => $string); if (-t) { binmode(STDIN, ":encoding(console_in)"); binmode(STDOUT, ":encoding(console_out)"); binmode(STDERR, ":encoding(console_out)"); } # Processing file names passed in as arguments my $uni_filename = decode(locale => $ARGV[0]); open(my $fh, "<", encode(locale_fs => $uni_filename)) || die "Can't open '$uni_filename': $!"; binmode($fh, ":encoding(locale)"); ... =head1 DESCRIPTION In many applications it's wise to let Perl use Unicode for the strings it processes. Most of the interfaces Perl has to the outside world are still byte based. Programs therefore need to decode byte strings that enter the program from the outside and encode them again on the way out. The POSIX locale system is used to specify both the language conventions requested by the user and the preferred character set to consume and output. The C module looks up the charset and encoding (called a CODESET in the locale jargon) and arranges for the L module to know this encoding under the name "locale". It means bytes obtained from the environment can be converted to Unicode strings by calling C<< Encode::encode(locale => $bytes) >> and converted back again with C<< Encode::decode(locale => $string) >>. Where file systems interfaces pass file names in and out of the program we also need care. The trend is for operating systems to use a fixed file encoding that don't actually depend on the locale; and this module determines the most appropriate encoding for file names. The L module will know this encoding under the name "locale_fs". For traditional Unix systems this will be an alias to the same encoding as "locale". For programs running in a terminal window (called a "Console" on some systems) the "locale" encoding is usually a good choice for what to expect as input and output. Some systems allows us to query the encoding set for the terminal and C will do that if available and make these encodings known under the C aliases "console_in" and "console_out". For systems where we can't determine the terminal encoding these will be aliased as the same encoding as "locale". The advice is to use "console_in" for input known to come from the terminal and "console_out" for output known to go from the terminal. In addition to arranging for various Encode aliases the following functions and variables are provided: =over =item decode_argv( ) =item decode_argv( Encode::FB_CROAK ) This will decode the command line arguments to perl (the C<@ARGV> array) in-place. The function will by default replace characters that can't be decoded by "\x{FFFD}", the Unicode replacement character. Any argument provided is passed as CHECK to underlying Encode::decode() call. Pass the value C to have the decoding croak if not all the command line arguments can be decoded. See L for details on other options for CHECK. =item env( $uni_key ) =item env( $uni_key => $uni_value ) Interface to get/set environment variables. Returns the current value as a Unicode string. The $uni_key and $uni_value arguments are expected to be Unicode strings as well. Passing C as $uni_value deletes the environment variable named $uni_key. The returned value will have the characters that can't be decoded replaced by "\x{FFFD}", the Unicode replacement character. There is no interface to request alternative CHECK behavior as for decode_argv(). If you need that you need to call encode/decode yourself. For example: my $key = Encode::encode(locale => $uni_key, Encode::FB_CROAK); my $uni_value = Encode::decode(locale => $ENV{$key}, Encode::FB_CROAK); =item reinit( ) =item reinit( $encoding ) Reinitialize the encodings from the locale. You want to call this function if you changed anything in the environment that might influence the locale. This function will croak if the determined encoding isn't recognized by the Encode module. With argument force $ENCODING_... variables to set to the given value. =item $ENCODING_LOCALE The encoding name determined to be suitable for the current locale. L know this encoding as "locale". =item $ENCODING_LOCALE_FS The encoding name determined to be suiteable for file system interfaces involving file names. L know this encoding as "locale_fs". =item $ENCODING_CONSOLE_IN =item $ENCODING_CONSOLE_OUT The encodings to be used for reading and writing output to the a console. L know these encodings as "console_in" and "console_out". =back =head1 NOTES This table summarizes the mapping of the encodings set up by the C module: Encode | | | Alias | Windows | Mac OS X | POSIX ------------+---------+--------------+------------ locale | ANSI | nl_langinfo | nl_langinfo locale_fs | ANSI | UTF-8 | nl_langinfo console_in | OEM | nl_langinfo | nl_langinfo console_out | OEM | nl_langinfo | nl_langinfo =head2 Windows Windows has basically 2 sets of APIs. A wide API (based on passing UTF-16 strings) and a byte based API based a character set called ANSI. The regular Perl interfaces to the OS currently only uses the ANSI APIs. Unfortunately ANSI is not a single character set. The encoding that corresponds to ANSI varies between different editions of Windows. For many western editions of Windows ANSI corresponds to CP-1252 which is a character set similar to ISO-8859-1. Conceptually the ANSI character set is a similar concept to the POSIX locale CODESET so this module figures out what the ANSI code page is and make this available as $ENCODING_LOCALE and the "locale" Encoding alias. Windows systems also operate with another byte based character set. It's called the OEM code page. This is the encoding that the Console takes as input and output. It's common for the OEM code page to differ from the ANSI code page. =head2 Mac OS X On Mac OS X the file system encoding is always UTF-8 while the locale can otherwise be set up as normal for POSIX systems. File names on Mac OS X will at the OS-level be converted to NFD-form. A file created by passing a NFC-filename will come in NFD-form from readdir(). See L for details of NFD/NFC. Actually, Apple does not follow the Unicode NFD standard since not all character ranges are decomposed. The claim is that this avoids problems with round trip conversions from old Mac text encodings. See L for details. =head2 POSIX (Linux and other Unixes) File systems might vary in what encoding is to be used for filenames. Since this module has no way to actually figure out what the is correct it goes with the best guess which is to assume filenames are encoding according to the current locale. Users are advised to always specify UTF-8 as the locale charset. =head1 SEE ALSO L, L =head1 AUTHOR Copyright 2010 Gisle Aas . This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself. =cut