perl: warning: Setting locale failed.

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

# locale-gen
# dpkg-reconfigure locales

List all installed Perl modules

# perl -MFile::Find=find -MFile::Spec::Functions -Tlwe 'find { wanted => sub { print canonpath $_ if /\.pm\z/ }, no_chdir => 1 }, @INC'

outputs a grid of 256 colors for checking your terminal in console

outputs a grid of 256 colors for checking your terminal in console

It should look like this:

256colors2.pl from http://www.frexx.de/xterm-256-notes/data/256colors2.pl

#!/usr/bin/perl
# Author: Todd Larason <jtl@molehill.org>
# $XFree86: xc/programs/xterm/vttests/256colors2.pl,v 1.2 2002/03/26 01:46:43 dickey Exp $

# use the resources for colors 0-15 - usually more-or-less a
# reproduction of the standard ANSI colors, but possibly more
# pleasing shades

# colors 16-231 are a 6x6x6 color cube
for ($red = 0; $red < 6; $red++) {
    for ($green = 0; $green < 6; $green++) {
 for ($blue = 0; $blue < 6; $blue++) {
     printf("\x1b]4;%d;rgb:%2.2x/%2.2x/%2.2x\x1b\\",
     16 + ($red * 36) + ($green * 6) + $blue,
     ($red ? ($red * 40 + 55) : 0),
     ($green ? ($green * 40 + 55) : 0),
     ($blue ? ($blue * 40 + 55) : 0));
 }
    }
}

# colors 232-255 are a grayscale ramp, intentionally leaving out
# black and white
for ($gray = 0; $gray < 24; $gray++) {
    $level = ($gray * 10) + 8;
    printf("\x1b]4;%d;rgb:%2.2x/%2.2x/%2.2x\x1b\\",
    232 + $gray, $level, $level, $level);
}


# display the colors

# first the system ones:
print "System colors:\n";
for ($color = 0; $color < 8; $color++) {
    print "\x1b[48;5;${color}m  ";
}
print "\x1b[0m\n";
for ($color = 8; $color < 16; $color++) {
    print "\x1b[48;5;${color}m  ";
}
print "\x1b[0m\n\n";

# now the color cube
print "Color cube, 6x6x6:\n";
for ($green = 0; $green < 6; $green++) {
    for ($red = 0; $red < 6; $red++) {
 for ($blue = 0; $blue < 6; $blue++) {
     $color = 16 + ($red * 36) + ($green * 6) + $blue;
     print "\x1b[48;5;${color}m  ";
 }
 print "\x1b[0m ";
    }
    print "\n";
}


# now the grayscale ramp
print "Grayscale ramp:\n";
for ($color = 232; $color < 256; $color++) {
    print "\x1b[48;5;${color}m  ";
}
print "\x1b[0m\n";

connect to mysql

C:\perl\bin\ppm.bat


Install:
- DBI
- DBD-MYSQL

Check:
perl -MDBD::mysql

split separate large file into small chunk files

split separate large file into small chunk files

#!C:\Perl\bin\perl.exe

foreach my $fileName (@ARGV) {
 $i = 0;

 open FILE, " open FILE2, ">C:/tmp/" . $fileName . $i or die $!;
 
 while () {
    # assigning each line to $str
   $str = $_;
   
   # 100000 lines per file
  if ( $i%100000 == 0 ) {
   $itmp = $i/100000;
   close FILE2;
   open FILE2, ">C:/tmp/" . $fileName . $itmp or die $!;
  }

  print FILE2 $str;

  $i++;
 }

 close FILE;
 close FILE2;
}

Using Perl's Finance::Quote to retrieve stock information from the Web

Using Perl's Finance::Quote to retrieve stock information from the Web

Looking for a way to add stock quotes to your web site? With a little Perl scripting and the Finance::Quote module you can script this up in no time.
Perl's CPAN contains many gems, and one of them is undoubtedly the Finance::Quote module. Designed to ease the task of retrieving stock prices off the Internet, the module makes it simple for developers to integrate stock-tracking mechanisms into their applications and thus create customised reports on financial market movements over differing time periods or investment portfolios.
If you're a Perl developer looking to build a Web-based stock ticker or portfolio tracker for equity markets and/or mutual funds, keep reading, because this article will teach you how to:
● Retrieve stock prices and market data for stocks traded on the major exchanges, such as the NYSE and NASDAQ.
● Retrieve mutual fund NAVs from a variety of online sources.
● Create a simple MySQL-based portfolio tracker to display the current value of an investment basket.InstallationThe Finance::Quote package was originally part of the GnuCash application until it was broken out as a separate project. The package consists of a base module that provides the underlying query methods and a series of supplementary modules that take care of parsing third-party Web sites for the required data. A variety of data sources are supported, and developers can easily add new sources as needed.
The package is licensed under the GPL and is currently maintained by Paul Fenwick. You can download it from the SourceForge site or from the CPAN page. The download archive provides detailed installation instructions, but by far the simplest way to install it is to use the CPAN shell:shell> perl -MCPAN -e shellcpan> install Finance::Quote
Finance::Quote depends on three other packages: LWP::UserAgent, HTTP::Request::Common, and HTML::TableExtractor. If yours is a relatively recent Perl installation, you'll probably already have the first two; you can download the third from this site. If you use the CPAN shell, dependencies will be automatically downloaded for you (unless you've configured your CPAN shell to ignore dependencies, in which case you'll need to download and install these three packages manually).
One important note before you proceed: The data retrieved by the Finance::Quote module is governed by the terms and conditions of the source from whence it was retrieved. So before you go out there and start selling your code to the highest bidder, take a close look at the license terms and data usage restrictions to ensure that you're in compliance. The examples here are purely illustrative. Okay, so let's start retrieving stock data.
Here's a simple example of using the Perl Finance::Quote package to retrieve the price of a fictitious stock (symbol XYZ) from the NYSE:#!/usr/bin/perl# import moduleuse Finance::Quote;# create objectmy $q = Finance::Quote->new();# retrieve stock quotemy %data = $q->fetch('nyse', 'XYZ');# print priceprint "The current price of XYZ on the NYSE is " . $data{'XYZ', 'price'};
Since Finance::Quote provides an object-oriented interface, the first step is to create an object with new(). This object exposes a number of methods, including the fetch() method. This method accepts two arguments: the name of the data source and the stock symbol.
The data returned by fetch() is structured as a two-dimensional hash: The first dimension is the stock symbol, and the second is a label corresponding to the attribute received. The hash returned by fetch() includes a number of different labels, some of which are listed below (a complete list is available in the online documentation):
nameâ€"official namepriceâ€"last tradehighâ€"highest price todaylowâ€"lowest price todayvolumeâ€"number of shares traded today year_rangeâ€"52-week price range peâ€"current P/E ratio navâ€"net asset value for mutual fund exchangeâ€"the source exchange successâ€"Boolean indicating if data retrieval was successful errormsgâ€"error string if data retrieval was unsuccessful
So, to retrieve a particular data fragment from the two-dimensional hash, you'd use the following notation:$hash(symbol, label)
This is clearly shown in the last line of the example above, which uses this notation to retrieve the price of XYZ stock.
Tip: You can have fetch() retrieve more than one quote at a time by specifying a comma-separated list (or an array) of stock symbols as the second input argument.
Okay, so that's stock prices. Now let's look at how to retrieve the NAV for a mutual fund.
The net asset value, or NAV, of a mutual fund is its assets minus its liabilitiesâ€"in essence, the net worth of the mutual fund. To retrieve a mutual fund's NAV using the Perl Finance::Quote package, simply use the fetch() method with an appropriate data source and the mutual fund symbol:#!/usr/bin/perl# import moduleuse Finance::Quote;# create objectmy $q = Finance::Quote->new();# set mutual fund code# each fund has a unique codemy $fcode = '1#761';# retrieve mutual fund NAVmy %quotes = $q->fetch('indiamutual', $fcode);# print detailsif ($quotes{$fcode, 'success'}){print "NAV of " . $quotes{$fcode, 'name'} . " is " . $quotes{$fcode, 'nav'}; } else{die "Data retrieval error";}
If Finance::Quote is unable to retrieve the data successfully, it sets the success key of the return hash to 0 (false) and places an error message in the errormsg key. By checking for this, you can implement basic error-handling routines into your Perl script, as I've done above.
Note that in many countries (for example, Canada and India), mutual funds do not have a unique global symbol. In such cases, you'll need to manually locate the mutual fund code assigned by the country's fund watchdog and use that in your Perl script. Links and instructions on how to accomplish this are included in the specific Finance::Quote submodule's documentation.
Now let's take a look at how your Perl program can retrieve the available data sources.

Since the Perl Finance::Quote package uses data from a variety of sources, a natural question (especially if you're letting the user choose the exchange or region) is: How do you find out which data sources are supported? The answer: with the sources() method. #!/usr/bin/perl# import moduleuse Finance::Quote;# create objectmy $q = Finance::Quote->new();# retrieve sourcesmy @sources = $q->sources();# print sourcesprint "Available sources are: ";foreach $s (@sources){print "$s ", ;}
Here's an example of the output:Available sources are: vwd nyse fidelity_direct canadamutual asxaustralia usa troweprice amfiindia tdwaterhouse fool nasdaqbmonesbittburns trustnet ftportfolios_direct yahoo_asia dutchuk_unit_trusts canada troweprice_direct tiaacref yahoo ftportfolios fidelity fundlibrary aex dwsfunds yahoo_australia yahoo_europe vanguard indiamutual asia europe
Some of the more important sources in this list are Yahoo! Finance (in all major world regions), the NYSE and NASDAQ, the Australian Stock Exchange, and data from Canadian, European, and Asian markets. In some cases, most notably Asia, you need to suffix the stock symbol with an exchange identifier as well.
You might notice an overlap among the various source names. This is because some are specific to exchanges, while others are more general country databases. For example, use "usa" for U.S. market data, but "nyse" or "nasdaq" to limit the query to the respective exchanges only.
Some of the more important sources in this list are Yahoo! Finance (in all major world regions), the NYSE and NASDAQ, the Australian Stock Exchange, and data from Canadian, European, and Asian markets. In some cases, most notably xxxx, you need to suffix the stock symbol with an exchange identifier as well.
You might notice an overlap among the various source names. This is because some are specific to exchanges, while others are more general country databases. For example, use "usa" for U.S. market data, but "nyse" or "nasdaq" to limit the query to the respective exchanges only.
With a little imagination, you can use the Perl Finance::Quote package for some very interesting financial applications. The one I like the most is an online portfolio tracker, which gives you real-time data on whether your investments are making money or not. Such an application would store information on your past stock purchasesâ€"quantity and purchase priceâ€"and retrieve, on demand or at regular intervals, the latest prices off the Web for quick comparison and evaluation.
To develop this application, I'll assume the existence of a MySQL table containing stock symbols, the quantities purchased of each, and the purchase prices of each. It looks something like this:
This example table holds portfolio information for a single user. Adding multiuser capability is as easy as adding a new owner field to each record and filtering by owner in the SQL query.
The next task is to develop a CGI script to communicate with MySQL, retrieve the portfolio data, and combine it with the latest prices from the Web. The original and current prices can then be compared to offer some insight into whether your portfolio is appreciating or depreciating, in both absolute and percentage terms (see Listing A below)
Code Listing A#!/usr/bin/perl # alter above to reflect the path to your Perl interpreter
# import modules
use Finance::Quote;
use DBI();
# print HTML headers
print "Content-type:text/html\n\n";
print "<head></head>";
# print table
print "<table border="1" cellpadding="5">";
print "<tr><td>Symbol</td><td>Quantity</td><td>Purchase
price</td><td>Current price</td> <td>Purchase value</td><td>Current value</td><td>Absolute gain/loss</td><td>% gain/loss</td></tr>\n";
# create Finance::Quote object
my $q = Finance::Quote->new();
# create DBI object
# connect
my $dbh = DBI->connect("DBI:mysql:database=stocks;host=localhost", "user",
"pass") or die("Error in connection");
# execute query
my $sth = $dbh->prepare("SELECT symbol, qty, pprice FROM portfolio"); $sth->execute();
# get everything as an array of arrays
my $result = $sth->fetchall_arrayref();
# iterate over record set
foreach $record (@$result)
{
# assign fields to variables
$symbol = $record->[0];
$qty = $record->[1];
$pprice = $record->[2];
# retrieve stock quote
my %data = $q->fetch('usa', $symbol);
$cprice = $data{$symbol, 'price'};
# calculate value
$pvalue = $pprice * $qty;
$cvalue = $cprice * $qty;
# calculate gain/loss in value
$diff = $cvalue - $pvalue;
$diff_percent = sprintf("%0.2f", $diff * 100/$pvalue);
# progressively calculate totals
$pvalue_total = $pvalue_total + $pvalue;
$cvalue_total = $cvalue_total + $cvalue;
# print table
print "<tr><td>$symbol</td><td>$qty</td><td>$pprice</td><td>$cprice</td>
<td>$pvalue</td><td>$cvalue</td><td>$diff</td><td>$diff_percent</td></tr>\n";
}
# calculate totals
$diff_total = $cvalue_total - $pvalue_total; $diff_total_percent = sprintf("%.2f", $diff_total * 100/$pvalue_total);
# clean up
$sth->finish();
$dbh->disconnect();
# print final row of totals
print "<tr><td></td><td></td><td></td><td></td>
<td><b>$pvalue_total</b></td><td><b>$cvalue_total</b></td>
<td><b>$diff_total</b></td><td><b>$diff_total_percent</b></td></tr>";
# close table
print "</table>";

After sending the standard HTML headers, the script in Listing A uses the Perl DBI to open a connection to the MySQL database and query it for the list of stocks, quantities, and original prices. Then, as it iterates over the record result returned by the query, the Finance::Quote module is used to retrieve the latest price of each stock. The two prices are then compared, and the change in absolute and percentage terms is recorded. Once all the records have been retrieved, the information generated is displayed in a neat table, together with summary totals.
Of course, this is just one of the myriad applications of Finance::Quote. With power like this at your fingertips, there's literally a whole world at your disposal. So go on...give your creativity full rein!

Grab / Retrieve / Crawling in Perl From web site web page - A Quick Tutorial

Grab / Retrieve / Crawling in Perl From web site web page - A Quick Tutorial
CS 594/494 Fall 2000
by John EblenGetting Started This tutorial shows you the basics of building Perl programs that can access and crawl the web, using Perl's libwww packages. To use these packages, you will need the following lines at the top of your program: use LWP::Simple;
use LWP::UserAgent;
use HTTP::Request;
use HTTP::Response;
use HTML::LinkExtor; # allows you to extract the links off of an HTML page.
In order to retrieve the contents of a page on the web, you can do the following: $contents = get($URL);
where URL is the URL of the page. $URL should be a full address, such as http://www.perl.com/. The entire contents of the page are then stored in $contents. That was pretty easy, wasn't it? Perhaps a little too easy, you might think, and you would be right. While this method works fine in most cases, it does not allow you to specify parameters for downloading, the most important one being a timeout value. Thus, any program using this method can become stuck on this particular line, waiting forever for a download that will never occur. By using a ctrl-c handler, you can rig the program to stop a download when ctrl-c is pressed, if you don't mind watching your program as it runs. Also, you can try to use a timeout signal, but that does not seem to work on our system (here at UT/CS). A Virtual Browser - a Better Way to Download Pages As you might have guessed, Perl provides a more sophisticated way to get pages. The following lines will initialize a virtual browser that can be used for all your browsing needs and will set the timeout to 10 seconds. Note that these packages take advantage of the object-oriented programming features provided by Perl, so the syntax may look strange to those who are new to Perl or to those who have never studied OO programming in Perl. $browser = LWP::UserAgent->new();
$browser->timeout(10);
Experimentation shows that the actual timing out usually takes a little longer than the amout given here, but at least the browser will time out. The next three lines will download the page indicated by $URL. my $request = HTTP::Request->new(GET => $URL);
my $response = $browser->request($request);
if ($response->is_error()) {printf "%s\n", $response->status_line;}
From an OO programming perspective, here's what happens. The first line creates a request object for $URL. The next line does the actual downloading of that request and returns a response object. The is_error() is a method of the object $response telling us whether or not there is an error, which if there were would be stored in the status_line instance variable of the response object. If you are not familiar with OO programming, you don't really need to understand all the vocabulary used in this paragraph in order to use these packages. Finally, the following retrieves the contents of the downloaded page: $contents = $response->content();
Extracting Links From a Web Page In order to do actual crawling, though, the above is not enough. You need to download web pages, but you also need to be able to further follow the links off of those pages. That is where the LinkExtor module comes in. The following code will take $URL and store the links of that URL into an array, @links: my ($page_parser) = HTML::LinkExtor->new(undef, $URL);
$page_parser->parse($contents)->eof;
@links = $page_parser->links;
Both $URL and $contents must be correct in order for this code to work, as it appends $URL to any relative links it finds in the page.
WARNING: The $URL should end in "/" in order for the appending to be done correctly. The following code will display all of the links found on the page: foreach $link (@links) {print "$$link[2]\n";}
Using What You've Learned With the following tools, you should now be able to build a simple but effective crawler in Perl. For any website you can begin at the main page for that site, download it, extract the links, and repeat the process with each of those links. You can continue until you retrieve no more links inside that domain. Of course there is an issue of exponential growth of links that have to be searched, but by using smart programming, you can control the process. Be sure to restrict the crawler to links inside the original domain, or you could potentially head out and start crawling the entire web! As you may have guessed, this tutorial provides an extremely sketchy, bare bones description of the above packages. Here, you have seen only what you would probably need for your projects and how to do only the most essential operations. You can find much more information inside the module files themselves that implement the packages. These module files end in .pm. So look for response.pm, request.pm, etc. Note: you can find the .pm files on the UTK/CS system at /mix/usr/local/lib/utk_perl5.

how to stop break loop for while

If Conditions
You've already seen if/elsif in action. The structure is always started by the word if, followed by a condition to be evaluated, then a pair of braces indicating the beginning and end of the code to be executed if the condition is true. The condition is enclosed in parentheses:
if (condition) {
code to be executed
}
The condition statement can be anything that evaluates to true or false. In Perl, any string is true except the empty string and 0. Any number is true except 0. An undefined value (or undef) is false.You can also test whether a certain value equals something, or doesn't equal something, or is greater than or less than something. There are different conditional test operators, depending on whether the variable you want to test is a string or a number:
Relational and Equality Operators
Test
Numbers
Strings
$x is equal to $y
$x == $y
$x eq $y
$x is not equal to $y
$x != $y
$x ne $y
$x is greater than $y
$x > $y
$x gt $y
$x is greater than or equal to $y
$x >= $y
$x ge $y
$x is less than $y
$x < $y
$x lt $y
$x is less than or equal to $y
$x <= $y
$x le $y
If it's a string test, you use the letter operators (eq, ne, lt, etc.), and if it's a numeric test, you use the symbols (==, !=, etc.). Also, if you are doing numeric tests, keep in mind that $x >= $y is not the same as $x => $y. Be sure to use the correct operator!
Here is an example of a numeric test. If $varname is greater than 23, the code inside the curly braces is executed:
if ($varname > 23) {
# do stuff here if the condition is true
}
If you need to have more than one condition, you can add elsif and else blocks:
if ($varname eq "somestring") {
# do stuff here if the condition is true
}
elsif ($varname eq "someotherstring") {
# do other stuff
}
else {
# do this if none of the other conditions are met
}
The line breaks are not required; this example is just as valid:
if ($varname > 23) {
print "$varname is greater than 23";
} elsif ($varname == 23) {
print "$varname is 23";
} else { print "$varname is less than 23"; }
You can join conditions together by using logical operators:Logical Operators
Operator
Example
Explanation
&&
condition1 && condition2
True if condition1 and condition2 are both true

condition1 condition2
True if either condition1 or condition2 is true
and
condition1 and condition2
Same as && but lower precedence
or
condition1 or condition2
Same as but lower precedence
Logical operators are evaluated from left to right. Precedence indicates which operator is evaluated first, in the event that more than one operator appears on one line. In a case like this:
condition1 condition2 && condition3
condition2 && condition3 is evaluated first, then the result of that evaluation is used in the evaluation.
and and or work the same way as && and , although they have lower precedence than their symbolic counterparts.
Unless
unless is similar to if. Let's say you wanted to execute code only if a certain condition were false. You could do something like this:
if ($varname != 23) {
# code to execute if $varname is not 23
}
The same test can be done using unless:
unless ($varname == 23) {
# code to execute if $varname is not 23
}
There is no "elseunless", but you can use an else clause:
unless ($varname == 23) {
# code to execute if $varname is not 23
} else {
# code to execute if $varname IS 23
}
Validating Form Data
You should always validate data submitted on a form; that is, check to see that the form fields aren't blank, and that the data submitted is in the format you expected. This is typically done with if/elsif blocks.
Here are some examples. This condition checks to see if the "name" field isn't blank:
if (param('name') eq "") {
&dienice("Please fill out the field for your name.");
}
You can also test multiple fields at the same time:
if (param('name') eq "" or param('email') eq "") {
&dienice("Please fill out the fields for your name
and email address.");
}
The above code will return an error if either the name or email fields are left blank.
param('fieldname') always returns one of the following:
undef — or undefined
fieldname is not defined in the form itself, or it's a checkbox/radio button field that wasn't checked.
the empty string
fieldname exists in the form but the user didn't type anything into that field (for text fields)
one or more values
whatever the user typed into the field(s)
If your form has more than one field containing the same fieldname, then the values are stored sequentially in an array, accessed by param('fieldname').
You should always validate all form data — even fields that are submitted as hidden fields in your form. Don't assume that your form is always the one calling your program. Any external site can send data to your CGI. Never trust form input data.
Looping
Loops allow you to repeat code for as long as a condition is met. Perl has several loop control structures: foreach, for, while and until.
Foreach Loops
foreach iterates through a list of values:
foreach my $i (@arrayname) {
# code here
}
This loops through each element of @arrayname, setting $i to the current array element for each pass through the loop. You may omit the loop variable $i:
foreach (@arrayname) {
# $_ is the current array element
}
This sets the special Perl variable $_ to each array element. $_ does not need to be declared (it's part of the Perl language) and its scope localized to the loop itself.
For Loops
Perl also supports C-style for loops:
for ($i = 1; $i < 23; $i++) {
# code here
}
The for statement uses a 3-part conditional: the loop initializer; the loop condition (how long to run the loop); and the loop re-initializer (what to do at the end of each iteration of the loop). In the above example, the loop initializes with $i being set to 1. The loop will run for as long as $i is less than 23, and at the end of each iteration $i is incremented by 1 using the auto-increment operator (++).
The conditional expressions are optional. You can do infinite loops by omitting all three conditions:
for (;;) {
# code here
}
You can also write infinite loops with while.
While Loops
A while loop executes as long as particular condition is true:
while (condition) {
# code to run as long as condition is true
}
Until Loops
until is the reverse of while. It executes as long as a particular condition is NOT true:
until (condition) {
# code to run as long as condition is not true
}
Infinite Loops
An infinite loop is usually written like so:
while (1) {
# code here
}
Obviously unless you want your program to run forever, you'll need some way to break out of these infinite loops. We'll look at breaking next.
Breaking from Loops
There are several ways to break from a loop. To stop the current loop iteration (and move on to the next one), use the next command:
foreach my $i (1..20) {
if ($i == 13) {
next;
}
print "$i\n";
}
This example prints the numbers from 1 to 20, except for the number 13. When it reaches 13, it skips to the next iteration of the loop.
To break out of a loop entirely, use the last command:
foreach my $i (1..20) {
if ($i == 13) {
last;
}
print "$i\n";
}
This example prints the numbers from 1 to 12, then terminates the loop when it reaches 13.
next and last only effect the innermost loop structure, so if you have something like this:
foreach my $i (@list1) {
foreach my $j (@list2) {
if ($i == 5 && $j == 23) {
last;
}
}
# this is where that last sends you
}
The last command only terminates the innermost loop. If you want to break out of the outer loop, you need to use loop labels:
OUTER: foreach my $i (@list1) {
INNER: foreach my $j (@list2) {
if ($i == 5 && $j == 23) {
last OUTER;
}
}
}
# this is where that last sends you
The loop label is a string that appears before the loop command (foreach, for, or while). In this example we used OUTER as the label for the outer foreach loop and INNER for the inner loop label.
Now that you've seen the various types of Perl control structures, let's look at how to apply them to handling advanced form data.

Perl Command-line Options

17 - Command-line Options
Perl has a wide range of command-line options or switches that you can use. The options are also called switches because they can turn on or turn off different behaviors. A thorough knowledge of the command line switches will enable you to create short one-time programs to perform odd little tasks. For example, the -e option lets you specify a line of code directly on the command-line instead of creating a script file. You use the -l option to change the line endings in a text file.
How Are the Options Specified?The most frequent way to specify a command-line options is on the command line. All of Perl's options are specified using a dash and then a single character followed by arguments, if needed. For example,
perl -I/usr/~john/include script.plYou can combine options with no arguments with the following switch. The following two command lines are equivalent.
perl -cI/usr/~john/include script.pl
perl -c -I/usr/~john/include script.plYou can also specify command-line options inside your script file using the #! line. Just place them following the directory and/or executable name. If you are working on a UNIX system, you are probably familiar with using the #! notation to tell the system where to find the Perl executable. The various UNIX systems and Windows can interpret the #! line in different ways. Therefore, Perl starts parsing the #! switches immediately after the first instance of "perl" on the line. For example, if you started your script with this line:
#!/bin/perl -wThen Perl will run with the -w option in effect.
Caution
Some UNIX systems will only read the first 32 characters of the #! line. So try to have your options either end before the 32nd position or start after the 32nd position. Placing the options after the 32nd position will help to make your scripts more portable because you will be bypassing one of the inconsistencies of UNIX.
What Are the Options?Table 17.1 provides a short description of each command-line option used with Perl. After the table, examples of several options will be shown.
Table 17.1-Perl's Command Line Options
Option
Description
-0
Lets you specify the record separator ($/) as an octal number. For example, - 0055 will cause records to end on a dash. If no number is specified, records will end on null characters. The special value of 00 will place Perl into paragraph mode. And 0777 will force Perl to read the whole file in one shot because 0777 is not a legal character value. See "Example: Using the -0 option" for more information.
-a
This option must be used in conjunction with either the -n or -p option. Using the -a option will automatically feed input lines to the split function. The results of the split are placed into the @F variable. See "Example: Using the -n and -p Options" for more information.
-c
This option lets you check the syntax of your script without fully executing it. The BEGIN blocks, and use statements are still executed because they are needed by the compilation process.
-d
This option lets you start the Perl debugger. See Chapter 16, "Debugging Perl," for more information.
-D
This option lets you turn on different behaviors related to the debugging process. The following table shows you the sub-options that can be used. Please note, however, that not all releases of Perl can use this feature. I know that the hip port of Perl for Win32 can't. If your version of Perl does not have this option, you will see the message Recompile perl with -DDEBUGGING to use -D switch when you try it. If you want to watch your script as it executes, use -D14. Following is a list of the other values that you can use. You can add the numbers together to specify more than one behavior (such as 8+4+2 = 14) or you can use the letters.
1
p
Tokenizing and Parsing
2
s
Stack Snapshots
4
l
Label Stack Processing
8
t
Trace Execution
16
o
Operator Node Construction
32
c
String/Numeric Conversions
64
P
Print Preprocessor Command for -P
128
m
Memory Allocation
256
f
Format Processing
512
r
Regular Expression Parsing
1024
x
Syntax Tree Dump
2048
u
Tainting Checks
4096
L
Memory Leaks (not supported anymore)
8192
H
Hash Dump -- usurps values()
16384
X
Scratchpad Allocation
32768
D
Cleaning Up
-e
This option lets you specify a single line of code on the command line. This line of code will be executed in lieu of a script file. You can use multiple -e options to create a multiple line program - although given the probability of a typing mistake, I'd create a script file instead. Semi-colons must be used to end Perl statements just like a normal script.
-F
This option modifies the behavior of the -a option. It lets you change the regular expression that is used to split the input lines. For example, -F /:+/ will split the input line whenever one or more colons are found. The slashes are optional; they simply delimit the pattern if they are there. I use them for their aesthetic value.
-i
This option lets you edit files in-place. It is used in conjunction with the -n or -p option. See "Example: Using the -i option" for more information.
-I
This option is used in conjunction with the -P option. It tells the C preprocessor where to look for include files. The default search directories include /usr/include and /usr/lib/Perl.
-l
This option turns on line-ending processing. It can be used to set the output line terminator variable ($/) by specifying an octal value. See "Example: Using the -0 option" for an example of using octal numbers. If no octal number is specified, the output line terminator is set equal to the input line terminator (such as $\ = $/;).
-n
This option places a loop around your script. It will automatically read a line from the diamond operator and then execute the script. It is most often used with the -e option. See "Examples: Using the -n and -p Options" for more information.
-p
This option places a loop around your script. It will automatically read a line from the diamond operator, execute the script, and then print $_. It is most often used with the -e option. See "Examples: Using the -n and -p Options" for more information.
-P
This option will invoke the C preprocessor before compiling your script. This might be useful if you have some C programming experience and would like to use the #include and #define facility. The C preprocessor can also be used for conditional compilation. Use the -I option to tell Perl where to find include files.
-s
This option lets you define custom switches for your script. See "Examples: Using the -s option" for more information.
-S
This option makes Perl search for the script file using the PATH environment variable. It's mostly used with UNIX systems that don't support the #! line. The docs/perlrun.htm documentation file that comes with your Perl distribution has more information about this option.
-T
This UNIX-based option turns on taint checking. Normally, these checks are only done when running setuid or setgid. The docs/perlsec.htm documentation file that comes with your Perl distribution has more information about this option.
-u
This UNIX-based option will cause Perl to dump core after compiling your script. See the Perl documentation that came with your Perl distribution for more information.
-U
This UNIX-based option will let Perl do unsafe operations. Its use is beyond the scope of this book.
-v
This option will display the version and patchlevel of your Perl executable.
-w
This option prints warnings about unsafe programming practices. See Chapter 16, "Debugging Perl," for more information.
-x
This option will let you extract a Perl script from the middle of a file. This feature comes in handy when someone has sent you a script via e-mail. Perl will scan the input file looking for a #! line that contains the word "perl". When it is found, it will execute the script until the __END__ token is found. If a directory name is specified after the -x option, Perl will switch to that directory before executing the script.
As you can see, Perl has quite a few command-line options. Most of them are designed so that you can do useful things without needing to create a text file to hold the script. If you are a system administrator then these options will make you more productive. You'll be able to manipulate files and data quickly and accurately. If you're looking to create applications or more complicated programs, you won't need these options - except for -w and -d.
The rest of the chapter is devoted to demonstrating the -0, -n, -p, -i, and -s options.
Example: Using the -0 OptionThe -0 option will let you change the record separator. This is useful if your records are separated by something other than a newline. Let's use the example of input records separated by a dash instead of a newline. First, you need to find out the octal value of the dash character. The easy way to do this is covert from the decimal value which will be displayed if you run the following command line.
perl -e "print ord('-');"This program will display 45. Converting 4510 into octal results in 558.
Next, you'll need an input file to practice with. Listing 17.1 shows a sample input file.
Listing 17.1-17LST01.DAT - Test Input File for the -0 Option
Veterinarian-Orthopedist-Dentist-
Listing 17.2 holds a program that reads the above data file using the diamond operators. The program will use the dash character as an end-of-line indicator.
Pseudocode
Set the record separator to be a dash using the #! switch setting method.
Open a file for input.
Read all of the records into the @lines array. One element in @lines will be one record.
Close the file.
Iterate over the @lines array and print each element.
Listing 17.2-17LST02.PL - Using the -0 Option to Change the Record Separator
#!perl -0055
open(FILE, "<17LST01.DAT");
@lines = ;
close(FILE);
foreach (@lines) {
print("$_\n");
}
Tip
Instead of using the command-line option, you could also say $/ = "-";. Using the command-line is a better option if the line ending changes from input file to input file.
This program will display:
Veterinarian-
Orthopedist-
Dentist-Notice that the end-of-line indicator is left as part of the record. This behavior also happens when the newline is used as the end-of-line indicator. You can use chop() or chomp() to remove the dash, if needed.
Example: Using the -n and -p OptionsThe -n and -p options wrap your script inside loops. Before looking at specific examples, let's see what the loops look like and how they are changed by the -a and -F options.
The -n option causes Perl to execute your script inside the following loop:
while (<>) {
# your script
}The -p option uses the same loop, but adds a continue block so that $_ will be printed every time through the loop. If both -n and -p are specified on the command line, the -p option will take precedence. The loop looks like this:
while (<>) {
# your script
} continue {
print;
}The -a option adds a split() function call to the beginning of each iteration of the loop. So that the loop looks like this:
while (<>) {
@F = split(/ /);
# your script
}The -F option lets you split on something besides the space character. If you used -F/+/ on the command line, the loop would look like this:
while (<>) {
@F = split(/+/);
# your script
}You can use BEGIN and END blocks if you need to specify some initialization or cleanup code. The initialization section might be used to create objects or to open log files. The cleanup section can be used to display statistics or close files. For example,
BEGIN {
# initialization section
$count = 0;
}
while (<>) {
# your script
}
END {
# cleanup section
print("The count was $count.\n");
}
Next, you'll see some examples of these options in action. Let's start with a command-line that simply displays each line of the input file - like the "type" command in DOS and UNIX.
The following examples use a data file called data.dat with the following lines: David Veterinarian
John Orthopedist
Jeff Dentist

perl -p -e "1;" data.datThis command line is equivalent to:
while (<>) {
1;
} continue {
print;
}
Note
The 1; statement was used to give Perl something to process. Otherwise, Perl would not have had any statements to execute.
And will display:
David Veterinarian
John Orthopedist
Jeff DentistHow about just printing the first word of each line? You could use this command line:
perl -p -e "s/\s*(\w+).*/$1/;" test.datwhich is equivalent to:
while (<>) {
s/\s*(\w+).*/$1/;
} continue {
print;
}And will display:
David
John
JeffIf you have data files that store information in columns, you can pull out the second column of information like this:
perl -p -e "s/\s*.+\s(.+)\s*/$1\n/;" test.datwhich will display:
Veterinarian
Orthopedist
DentistYou can use the -a option to get access to information stored in columns. For example, you could also display the second column like this:
perl -p -a -e "$_ = \"$F[1]\n\";" test.datwhich is equivalent to
while (<>) {
@F = split(/ /);
$_ = "$F[1]\n";
} continue {
print;
}
Notice you need to escape the double-quotes in the above command-line. If you don't do this you get an error message.
Example: Using the -i OptionThe -i option lets you modify files in-place. This means that Perl will automatically rename the input file and open the output file using the original name. You can force Perl to create a backup file by specifying a file extension for the backup file immediately after the -i. For example, -i.bak. If no extension is specified, no backup file will be kept.
One of the more popular uses for the -i option is to change sequences of characters. This kind of change normally requires 10 or more lines of code. However, using command line options you can do it like this:
perl -p -i.bak -e "s/harry/tom/g;" test.datThis command line will change all occurrences of "harry" to "tom" in the test.dat file.
Examples: Using the -s OptionThe -s option lets you create your own custom switches. Custom switches are placed after the script name but before any filename arguments. Any custom switches are removed from the @ARGV array. Then a scalar variable named after the switch is created and initialized to 1. For example, let's say that you want to use a switch called -useTR in a script like the one in Listing 17.3.
Listing 17.3-17LST03.PL - Checking for the useTR Switch
if ($useTR) {
# do TR processing.
print "useTR=$useTR\n";
}
You might execute this program using this following command line:
perl -s -w 17lst03.pl -useTRand it would display:
useTR=1
SummaryThis chapter covered the different command-line options that you can use with Perl. The options can also be referred to as switches because they turn different behaviors on and off.
The switches can be specified on the command line or using the #! line inside your script. If you use the #! line, try to place the options after the 32nd position to avoid inconsistent handling by different versions of UNIX.
The -n option is used to place your script inside of an input loop. The -p option uses the same loop, but also prints the $_ variable after each pass through the loop. The -a and -F options are used when you want the input lines to be split into the @F array.
Another very useful option is -i, which lets you edit files in-place. This option is good when you are doing a lot of text file manipulation.
The next chapter, "Using Internet Protocols," introduces you to some of the different standards used on the Internet. These standards let you perform activities like read mail, send mail, and transfer files.
Review Questions
What is a command line option?
What are the two places that the switches can be specified?
What switch should always be used?
Which switch lets you read records that end with the ~ character instead of the newline?
What two options can be used with the -n option?
How can you execute a script that someone sent you via E-mail?
What happens if you specify both the -v and the -c options?
Review Exercises
Use the -v option to see the patchlevel of your version of Perl.
Use the chomp or chop function to remove the dash from the end of the records printed by the program listing 17.2.
Write a program that uses the -p option to display the third column.
Modify the program written in exercise 3 to use a BEGIN block to ask the user which column to display?
Create a sample E-mail message that contains a Perl script. Use the -x option to execute it.
Modify the E-mail message written for exercise 5 to display any text that appears after the __END__ token. Hint: Use the DATA file handle.

Perl Regular Expressions

Perl Regular Expressions
In Perl, you write regular expressions between / delimiters (or you can change the delimiter if you wish), and you add modifiers after the closing /. To match the contents of a variable to a regular expression, use the =~ operator. Regular expressions are also used by perl built in functions such as grep and split, and by the s operator.Perl uses a very full set of elements within its regular expressions, most of which are terse so hard for the newcomer to follow when maintaining code. It predates, so does not follow, the POSIX standard.Perl 6, currently under development, will support grammars and rules rather than regular expressions. Grammars and Rules will take pattern matching to a whole new level, and tools will be available to covert code - in other words, rules and grammars will do everything that the old Regular Expressions didn't, and more.
Operator Type
Examples
Description
Literal CharactersMatch a character exactly
a A y 6 % @
Letters, digits and many specialcharacters match exactly
\$ \^ \+ \\ \?
Precede other special characterswith a \ to cancel their regex special meaning
\n \t \r
Literal new line, tab, return
\cJ \cG
Control codes
\xa3
Hex codes for any character
Anchors and assertions
^
Starts with
$
Ends with
\b \B
on a word boundary,NOT on a word boundary
Character groupsany 1 character from the group
[aAeEiou]
any character listed from [ to ]
[^aAeEiou]
any character except aAeEio or u
[a-fA-F0-9]
any hex character (0 to 9 or a to f)
.
any character at all(not new line in some circumstances)
\s
any space character (space \n \r or \t)
\w
any word character (letter digit or _)
\d
any digit (0 through 9)
\S \W \D
any character that is NOT a spaceword character or digit
Countsapply to previous element
+
1 or more ("some")
*
0 or more ("perhaps some")
?
0 or 1 ("perhaps a")
{4}
exactly 4
{4,}
4 or more
{4,8}
between 4 and 8
Add a ? after any count to turn it sparse (match as few as possible) rather than have it default to greedy
Alternation

either, or
Grouping
( )
group for count and save to variable
(?: )
group for count but do not save
Variables
$xyz
Insert contents of $xyz into regular expression
\1 \2
Back reference to 1st, 2nd etc matched groupsAfter the closing / of your regular expression, you can add one or more modifiers to change its behaviour.
Modifier
Description
i
Ignore case in matching
g
Global match. Return a list of all matches (list context) or return the next match (scalar context)
x
White space is to be treated as a comment (otherwise it matches exactly)
s
. to match everything including new line (otherwise it matches everything except new line)
m
^ and $ to match embedded new lines
o
Tell compiler that regular expression doesn't change even if it includes a variable reference
e
s command only. Execute the output before you substitute it inThe following Perl functions and operators use regular expressions
Function / Operator
use

If you write a regular expression without an operator, it matches the regular expression against the contents of the $_ variable.
=~
Match the regular expression to the right against the variable to the left
s
Substitute the matched regular expression with a replacement string
grep
Filter a list for all member scalars that match the regular expression
split
split a scalar into a list, dividing the elements at the regular expressionThe above lists show the most commonly used elements of Perl regular expressions, and are not exhaustive.In Perl, you can change the / regular expression delimiter to almost any other special character if you preceed it with the letter m (for match); if you change to ( { or [, the balancing end expression character becomes ) } or ]. Back to Regular Expression Home PageJump to Elements of a regular expressionOrder a Regular Expression Mousemat for £4.95 inclusive

Remove Smart Quote Quotation / weird unwanted character from a CSV file (Comma separated values)

Remove Smart Quote Quotation / weird unwanted character from a CSV file (Comma separated values)

Today, I was using Regular Expression to parse the data. I encountered a weird character problem, which is so-called the smart quote. The smart quote messes up my rest of data, makes the parser not able to parse the data correctly. Then, I start looking for a method to edit hex in vi / vim . This is what I found:

method 1:
Type following command in the vi's command mode to convert the file to a display in hex.
:%!xxd

Type following command when you are done in hex.
:%!xxd -r

method 2:
start out with vi -b filename to edit a hex or binary file,

method 3:
Type following command in the vi's command mode
:set display=uhex.

The hex code of smart quote can be one of following depends on which encoding type you used:

windows-1252: 93
iso-8859-1: 3F (question mark, i.e., cannot be represented)
Big5: A1A7
UTF-8: E2809C
UTF-16: 201C
UTF-16BE: 201C
UTF-16LE: 1C20
UTF-7: /v8gHA (not the hex representation, that would be: 2F7638674851)
Shift-Jis: 8167
IBM500: 3F (wrongly represented by the serializer: SUB is a control code in IBM encodings)
GB 18030: A1B0
MacRoman: D2 (the mac codepage actually has a 'smart quote')
EUC-JP: A1C8

Finally, I come up with a batch script to replace these smart quotes to # (23) number sign:

Note: I saved the file as the Western European (Windows) encoding (windows-1252), so the code of smart quote is 93 (left quote) and 94 (right quote)

Recently I’ve been running into a lot of silliness that appears in files exported from MS applications (Word, Excel, etc.) called ‘Smart quotes’.

Basically MS uses higher level ascii characters to represent quotes that mean more than regular quotes (whatever for?!). You see this weirdness in vi as <93><92> etc. which are the hex values of these characters. I had to hunt a bunch on google to find out how to fix this, although the fix is very easy.

For each value that you see in your file, just do a string substitution, like so:

:%s/<93>/\’/g

of course, you can’t just type that <93> in there, so to get it in there you use (via: http://www.vim.org/htmldoc/usr_45.html)

CTRL-V x 93

which inserts hex 93 in place.

In recently exported CSV’s from excel, I’ve seen hex 91-97.

Quite annoying, frankly.

–

You could also use perl, like here

On Unix:
perl -pi -e 's/\x93\x94/"/g' myfile.html

On Windows:
perl -pi -i.bak -e "s/\x93\x94/\"/g" myfile.html

and use it on multiple files.

removeChars.pl:

#!C:\Perl\bin\perl.exe

# this script does four things:
# 1) remove the first line.
# 2) replace all commas inside quotes to underscore (_)
# 3) Remove all quotes
# 4) Remove the lines that its first field is not digit.
# Run: perl parse.pl file_name

#$fileName = @ARGV[0];
#printf("\n");

foreach my $fileName (@ARGV) {
 open FILE, " open FILE2, ">zip_file/data/" . $fileName . ".done" or die $!;

 $i = 0;
 while () {
   # I don't need the first line, so we start from second line.
  if ($i==1) {
    # assigning each line to $str
   $str = $_;

   $stopLoop = ( $str =~ m/^[^0]/ );
   if ($stopLoop) {
    last;
   }

   # $str=~ s/("[^"]+")/remove($1,$1)/eg; # for OO
   $str=~ s/("[^"]+")/remove($1)/eg;
    # Note, if you want to use your own defined function inside 
    # the s (substitution) command (remove() in above codes for example). 
    # You will have to use the e Modifier
    # e: s (substitution) command only. Execute the output before you substitute it in 

    # write $str to file2
   print FILE2 $str
  } else{
   $i = 1;
  }
 }

 close FILE;
 close FILE2;
}



#$str = '123, "how, are, you", "i, am", "fine, thank, you, and, you"';
#$str=~ s/("[^"]+")/remove($1)/eg;
#print $str;


sub remove {
 # my( $self, $s ) = @_; # this is for object orient
 my $s = shift or return;

 # find: quote, comma, smart quote (left), smart quote (right)
 # replace: nothing
 # note: can use {$1} to separate other text
 $s =~ s/"|,|\x93|\x94//g;
 return $s;
}


#分小塊讀入文件到緩存，然後寫出。優點:服務器系統資源損耗小，客戶端下載速度快。
# while(sysread(DLFILE, my $buffer, 256)){print $buffer;}
# close (DLFILE) || $main->printMessage('關閉文件錯誤，可能文件不存在。出現此錯誤請和網管聯繫，謝謝。');


sub de_cp1252sub de_cp1252 {
  my( $self, $s ) = @_;

  #   Map incompatible CP-1252 characters
  $s =~ s/\x82/,/g;
  $s =~ s-\x83-<em>f</em>-g;
  $s =~ s/\x84/,,/g;
  $s =~ s/\x85/.../g;

  $s =~ s/\x88/^/g;
  $s =~ s-\x89- °/°°-g;

  $s =~ s/\x8B/</g;
  $s =~ s/\x8C/Oe/g;

  $s =~ s/\x91/'/g;
  $s =~ s/\x92/'/g;
  $s =~ s/\x93/"/g;
  $s =~ s/\x94/"/g;
  $s =~ s/\x95/*/g;
  $s =~ s/\x96/-/g;
  $s =~ s/\x97/--/g;
  $s =~ s-\x98-<sup>~</sup>-g;
  $s =~ s-\x99-<sup>TM</sup>-g;

  $s =~ s/\x9B/>/g;
  $s =~ s/\x9C/oe/g;

  #   Now check for any remaining untranslated characters.
  if ($s =~ m/[\x00-\x08\x10-\x1F\x80-\x9F]/) {
      for( my $i = 0; $i < length($s); $i++) {
          my $c = substr($s, $i, 1);
          if ($c =~ m/[\x00-\x09\x10-\x1F\x80-\x9F]/) {
              printf(STDERR  "warning--untranslated character 0x%02X i
+n input line %s\n",
                  unpack('C', $c), $s );
          }
      }
  }

  $s;
}

Reference:
http://aditya.sublucid.com/2008/01/18/replacing-those-pesky-smart-quotes-in-vim/
http://www.usenet-forums.com/linux-general/98618-can-vi-show-edit-binary.html
http://www.biglist.com/lists/xsl-list/archives/200709/msg00358.html
http://lists.freebsd.org/pipermail/freebsd-questions/2003-July/012019.html


Re: [xsl] RE: Smart Quote Encoding
Subject: Re: [xsl] RE: Smart Quote Encoding
From: Abel Braaksma
Date: Fri, 14 Sep 2007 03:17:15 +0200


Hi Roger,

see below



Roger L. Cauvin wrote:
messages to XML. For various reasons (including troubleshooting), I would
like to log the content of the e-mails exactly. It sounds like that's
simply not possible in XML, at least to the extent that "text-only" can
include characters not allowed in XML.
I have a program that is receiving text-only e-mails and logging the


Text only is something that is very much misunderstood, simply because there ain't such a thing as text-only. Is that IBM-437 format (that is: original DOS). Is that ASCII? Some people call it plain old ascii when it is in fact windows-1252. And many people don't know the differences between windows-1252 and ISO-8859-1. If text-only means UTF-8, it will contain a lot of "binary" looking bytes if viewed with windows-1252 encoding. If you have a mail from gmail, it will be send in UTF-7 (yes, I know, strange). Which is yet another binary format for text-only.

And last but not least, if you have text-only in EBCDIC encoding, it will look like a mess in Windows. But luckily, XML can handle all these encodings, but parsers are only required to handle UTF-8 and UTF-16. Saxon, in my experience, can handle ebcdic well, but not UTF-7 (but that's to blame with Sun who's been refusing for ages to include UTF-7).


I inserted the ISO 8859-1 encoding declaration myself. Apparently, Saxon
6.3 doesn't support windows-1252 encoding. Saxon 8.9J, which I just now
installed, does appear to support that encoding. However, it still
(correctly) flags the U+18 character as illegal.


that does not depend on Saxon but on the Java version you use. Btw, why would you want to use an age-old version of Saxon? Saxon 8.9 can do XML 1.1 and that can represent character U+18.

But you should workout what the encoding is your email program is using. It may be that it stores the file in the same encoding as it is received, but it is more likely that the email program transforms it into some other format. If you read your email from the interface of the email program (i.e., Thunderbird) then you will see what is done with the code. But when you view your email in bare format, you'll have to find out what the email program does to it. In the case of Thunderbird, I believe it stores all in one file and encodes it as UTF-8, but I am not sure (and you probably use a different mailer). Maybe TB even stores different text formats in one file...


More about U+18
---------------
Unfortunately, without knowing what the byte sequence of that character is and without a binary view of your whole file, it is quite hard to determine what the real encoding is. Like David already suggested: you should check the encoding in your file.

Furthermore, you are dealing with a lot of text it seems. If you want more control over what encoding you can choose, you can try unparsed-text($url, $encoding) in XSLT 2.0. Note that you must remove the XML declaration then, because the spec states that the encoding in the declaration has higher precedence than the user specified encoding.

If you want to check a bunch of encodings all at once and see if one fits, you can lookup the list of supported encodings at Sun's website. Make it into a sequence, i.e. and you can loop through all possibilities using:



But to me, it sounds like you have encountered a mail in the lesser used UTF-7 encoding. You were talking of the windows quote. Let's examine that (assuming that you are correct in your analysis that the quote appears where you have found U+18 (which is not U+18, but a byte with hex value 18 that does not translate to the correct character in the encoding that you guessed the file was in)), and this 18h is part of a longer byte sequence where the smart quote is used. The smart code, I hope, is ” (or &_#x201D if the browser/mailer screws it up), which is the one MS Word uses.

You read the file using 'ISO-8859-1'. That means that the first 127 bytes equal there counterparts in the Unicode table (i.e. U+0 to U+7F are the same as the bytes in the stream).

Now, let's see how the smart quote really should look like in certain encodings that I think are likely to be encountered in mail (alle bytes zijn in hex):

windows-1252: 93
iso-8859-1: 3F (question mark, i.e., cannot be represented)
Big5: A1A7
UTF-8: E2809C
UTF-16: 201C
UTF-16BE: 201C
UTF-16LE: 1C20
UTF-7: /v8gHA (not the hex representation, that would be: 2F7638674851)
Shift-Jis: 8167
IBM500: 3F (wrongly represented by the serializer: SUB is a control code in IBM encodings)
GB 18030: A1B0
MacRoman: D2 (the mac codepage actually has a 'smart quote')
EUC-JP: A1C8


I got this information by combining saxon:serialize, saxon:string-to-hexBinary and saxon:string-to-base64Binary (the last one for creating the not supported and non-unicode standard UTF-7). As you can see, there is no encoding in existence that has ever used byte 18 to represent a high character.

The only situation that I can think of to legally have a byte 18 in a sequence of bytes to represent a character, is in multi byte encoding formats. I.e., UTF-16, GB 18030, Big5 etc. I believe that, because it is a control character, that you will only find it as the second byte in any two byte sequence and never as the first byte.


All this just for one character that is illegally encoded in the source? Well, email programs, like I said, save their data in a variety of formats. Or set the output options to XML 1.1 and use unparsed-text.

Hope this "little" story clarifies things a bit for you.


cheers,
-- Abel Braaksma

vi vim Remove Replace comma commas inside quotes double quotation Regular Expression CSV Comma separated values

vi vim Remove Replace comma commas inside quotes double quotation Regular Expression CSV Comma separated values

Wow, I finally made it! It took me few days of night and coffee to come up this solution. This vi / vim Regular Expression can replace / substitute the commas inside quotes (substitute inside substitute). CSV Comma separated values

Sample Text:

"Hello,World", "vi, vim, gvim, is a really fun editor"
"I am really, really, really, excited", "it,made,my,day."

Replace Commas Inside Quotes:

%s/"\([^"]\+\)"/\=substitute(submatch(0), ',', '_', 'g')/g

Sample Output:

"Hello_World", "vi_ vim_ gvim_ is a really fun editor"
"I am really_ really_ really_ excited", "it_made_my_day."

Run it in batch script:

@echo off

vim -S replace5.vim test.txt

:: http://gala4th.blogspot.com/2008/12/vi-vim-remove-replace-comma-commas.html

echo Done!
PAUSE

replace5.vim:

:set nobackup
:argdo! %s/"\([^"]\+\)"/\=substitute(submatch(0), ',', '_', 'g')/g  update
:wq

Final Version:

@echo off

:: replace7.vim does three things:
:: 1) remove the first line.
:: 2) replace all commas inside quotes to underscore (_)
:: 3) Remove all quotes
vim -S replace7.vim test.txt

:: http://gala4th.blogspot.com/2008/12/vi-vim-remove-replace-comma-commas.html

echo Done!
PAUSE

replace7.vim

:set nobackup
:argdo! 1,1s/^.\+\n//
:argdo! %s/"\([^"]\+\)"/\=substitute(submatch(0), ',', '_', 'g')/g
:argdo! %s/"//g | update
:wq

Perl Version:

#!C:\Perl\bin\perl.exe

$str = '123, "how, are, you", "i, am", "fine, thank, you, and, you"';
$str=~ s/("[^"]+")/remove($1,$1)/eg;
print $str;

sub remove {
 my( $self, $s ) = @_;

 # find: quote, comma, smart quote (left), smart quote (right)
 # replace: nothing
 # note: can use {$1} to separate other text
 $s =~ s/"|,|\x93|\x94//g;
 return $s;
}

See Also: http://gala4th.blogspot.com/2008/12/vi-vim-gvim-find-search-replace-string.html

Reference:
http://www.nminus.org/vim/
http://www.geocities.com/volontir/
http://www.guckes.net/vi/substitute.html
http://vimdoc.sourceforge.net/htmldoc/change.html
http://www.vim.org/htmldoc/pattern.html
http://vim.wikia.com/wiki/Using_an_expression_in_substitute_command
http://www.jason.mock.ws/wordpress/2005/10/28/vim-tips