Re: Create a pg table from CSV with header rows

[David Fetter <david@xxxxxxxxxx>]

On Fri, Sep 16, 2005 at 10:41:59AM -0400, Doug Bloebaum wrote:
> On 9/15/05, Michael Fuhr <mike@xxxxxxxx> wrote:
> > 
> > On Thu, Sep 15, 2005 at 06:48:59PM -0400, Robert Fitzpatrick wrote:
> > > Anyone know a package that can do this? Perferrably a Unix/Linux
> > > package.
> > 
> > It would be trivial to write a script in a language like Perl to
> > read the first line of a file and generate a CREATE TABLE statement
> > from it, then issue a COPY command and send the rest of the file.
> > Determining the columns' data types would be a different matter:
> > if they weren't specified in the header then you'd have to guess
> > or perhaps make them all text.
> 
> 
> I've attached a sample implementation of Michael's suggestion.

I've attached what I hope is a slightly improved one :)

Cheers,
D
-- 
David Fetter david@xxxxxxxxxx http://fetter.org/
phone: +1 510 893 6100   mobile: +1 415 235 3778

Remember to vote!

#!/usr/bin/perl -w

use DBI;
use Text::CSV_XS;
use File::Basename;

my $csv_file = shift or die "Usage: $0 <file.csv>\n";
my $fh; # Filehandle.
open $fh, '<', $csv_file # 3-argument open in case of name weirness.
    or die "$0: Can't open $csv_file for read: $!\n";

my ($table_name) = fileparse($csv_file,qr{\..*});
$table_name = quote_odd_name($table_name);
warn "table_name=$table_name\n";

my $dbh = DBI->connect(
  'dbi:Pg:dbname=dbname', 'user', 'pass', # should this be user input?
  { Auto_commit => 0 }
);
my $csv = Text::CSV_XS->new();

# create the table based on names from the header row
# columns will arbitrarily be of type TEXT, because VARCHAR(20)
# is a pretty silly limitation.

my $header = <$fh>;
$csv->parse($header) or die "$0: parse of header row failed\n";
my @columns = $csv->fields();
my $sql = "CREATE TABLE $table_name (\n  "
        . join(",\n  ", map {quote_odd_name($_).' TEXT'} @columns)
        . "\n)";
warn "\n$sql\n";
$dbh->do($sql);

# add rows of data using COPY
$dbh->do("COPY $table_name FROM STDIN WITH DELIMITER AS ','");
my $row_count = 0;
while(<$fh>) {
  $dbh->pg_putline($_);
  $row_count++;
}
close $fh;
$dbh->pg_endcopy;
$dbh->commit;
$dbh->disconnect;

warn "$0: Added $row_count rows to $table_name\n";

sub quote_odd_name {
    my $name = shift;
    my $sth = $dbh->prepare_cached('SELECT quote_ident(?)');
    $sth->execute($name);
    my $row = $sth->fetchrow_hashref;
    $sth->finish;
    return $row->{quote_ident};
}

---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings