#!/usr/bin/perl -w

=pod

=head1 NAME

tv_grab_na - Grab Canadian and US TV listings.

=head1 SYNOPSIS

tv_grab_na [--help] [--debug] [--config-file FILE] [--output FILE] --configure CONFIG_OPTIONS

tv_grab_na [--help] [--debug] [--config-file FILE] [--output FILE] GRAB_OPTIONS

tv_grab_na [--output FILE] (--postalcode X | --zipcode X) --list-providers

tv_grab_na [--output FILE] (--postalcode X | --zipcode X) --provider P --list-channels

=head1 DESCRIPTION

Output TV listings for Canada and the USA in XMLTV format.  The data
currently comes from the zap2it.com site.

First you must run B<tv_grab_na --configure> to set your local area,
service provider and which channels you wish to download.  Then
running B<tv_grab_na> with no arguments will get a weekE<39>s listings
for the channels you chose.  If the upstream data changes (for
example, new channels appear) you will be prompted to rerun
B<--configure> to update your choices.

Some options are available in both configure and grab mode:

B<--help> Print help to standard output and exit successfully.

B<--debug> Turn on debugging checks.

B<--quiet> disable all status messages (that normally appear on stderr).

B<--release-check X> Specify whether or not we should attempt to check
for new releases. X should be either "true" or "false"
at http://sourceforge.net/projects/xmltv (default=true)
(output disabled if --quiet is used)

B<--config-file FILE> The location of the config file, which is
written in configure mode and read in grab mode.  The default location
is B<~/.xmltv/tv_grab_na.conf>.

B<--output FILE> Write output to FILE rather than standard output.

Configuration is normally interactive, but it can be run
noninteractively, usually to update an existing set of choices. The
following options are used to invoke --configure in a non-interactive
mode:

B<--retry-limit X> Specify retry limit on www site failures.

B<--retry-delay X> Specify number of seconds to delay before retrying
www site failures.

B<--postalcode X>, B<--zipcode X> Specify your postal code or zip code.

B<--provider ID> Specify the id of your service provider.  This is a
number assigned by the zap2it site.  During interactive configuration
you are presented with a list of provider names, but if you already
know the id you can specify it here.

B<--auto-fail-on-provider-changes BOOLEAN> If true, exit with failure
if the provider id has changed on the zap2it site.

B<--auto-new-channels [ignore|add]> Ignore or automatically add new
channels in schedule listing.

B<--auto-missing-channels [ignore|remove]> Ignore or remove channels
no longer in schedule listing.

Interactive mode is entered if some necessary information is not given
(for example no postal or zip code in the existing configuration file,
and not given on the command line).

After successful configuration you can start grabbing.  The options
here are:

B<--debuglistings> Add debugging information to the output as XML
comments.

B<--stats> force output of grab stats (stats output disabled in --quiet mode).

B<--output FILE> Write output to FILE rather than standard output.
If the file exists it is overwritten. FILE may contain L<date(1)>
and keyword substitions like B<--listings>. 

B<--listings FILE> Write output to FILE rather than standard output.
FILE may contain L<date(1)> style substitutions, for example use
B<--listings "listings-%d%m%Y.xml"> to separate output by day.
Keywords B<%postalcode> and B<%zipcode> are also substituted from
settings current configuration. Channel-based keywords B<%ChannelNumber> and
B<%ChannelCallLetters> may be used, which cause the filename to be
re-evaluated between each channel for each day, instead of just when a
new days listings is started. Be warned, adverse results can occur if you
use the B<%ChannelNumber> and/or B<%ChannelCallLetters> in the output
pathname for a multi-day grab without some %d (or similar) substitution.
For instance, if you use --listings "listings-%ChannelNumber" for a
multi-day grab, the listings file will only contain the first days
listings (or last days listings if --listings-overwrite is used)
since the file will be re-evaluated after ever channel grab.

B<--listings-overwrite X> Should tv_grab_na overwrite existing --listings output
files or skip the ones that exist? (default=false - do not overwrite
already grabbed listings files)

B<--days N> When grabbing, grab N days of listings.  The default is 7.

B<--offset N> Number of days in the future to offset the start of the
listings.  The default is 0, meaning today.

B<--retry-limit X> Specify retry limit on www site failures.

B<--retry-delay X> Specify number of seconds to delay before retrying
www site failures.

B<--gzip-command X> Specify the command line for .gz output files.
The default is "gzip".

B<--zip-command X> Specify the command line for .zip output files.
The default is "zip".

B<--bzip2-command X> Specify the command line for .bz2 output files.
The default is "bzip2".

To help programs that want to do automated configuration there are the
options:

B<--list-providers> Produce a list of providers and their descriptions
for the postal/zip code provided in --postalcode or --zipcode arguments.

B<--list-channels> Produce a list of channels available. Requires you
also specify provider/location information via --provider and one of
--postalcode or --zipcode options.

=head1 SEE ALSO

L<xmltv(5)>, L<http://www.zap2it.com/>

=head1 AUTHOR

Jerry Veldhuis, jerry@matilda.com

=head1 BUGS

Zap2it recently removed the (ends at HH:MM) hints in their
text listings. This value was being use to confidently compute
the programme's stop time, without this, we no long can. Instead
we compute the stop time by assuming it stops when the next
program starts. This doesn't work for the last program of the
the day (since the listings only show one day at a time and
we grab chronologically), for these programs, tv_grab_na doesn't
emit a stop time in the programme in the xml output. A post-grab
call to tv_sort can fill these in using the next programme start
time, but this assumes the next program is always available.
WARNING: Although zap2it seems to have fixed the programming holes
problem, this format change removes our chances of confidently
finding them.

Because we parse the text listings (at zap2it), not the grid,
output doesn't include the colour coded program categories
such as Sports,Movie,Series,News,Special and Children that
appear in the grid listings. 

There is no way to get listings for only part of each day; the old
B<--startHour> and B<--endHour> options no longer exist.  This makes
grabbing slower than it needs to be if you only want listings for
certain hours.

The zap2it site has a long list of program qualifiers such as Live,
Animated, HDTV and so on.  Most of these are understood by tv_grab_na
and included in the XMLTV output.  But some new ones pop up
occasionally and generate a warning message (which should be reported
to the author).  Some qualifiers such as Live do not yet have a
good translation into XMLTV format.

=cut

use strict;
use XMLTV::Version '$Id: tv_grab_na,v 1.98 2004/03/02 21:00:03 epaepa Exp $ ';
sub checkCache();

my $VersionMajor=3;
my $VersionMinor;

my $Date = q$Date: 2004/03/02 21:00:03 $;
if ( !($Date=~m;^\s*Date: (\d{4})/(\d\d)/(\d\d) \d\d:\d\d:\d\d\s*$;)) {
    die "can't decipher date line ($Date)\n";
}
else {
    # Take the day as the minor version, eg 20020224.
    $VersionMinor="$1$2$3";
}

my $VersionID="tv_grab_na V$VersionMajor\.$VersionMinor";

package myConfig;

sub new
{
    my($type) = shift;
    my $self={ @_ };            # remaining args become attributes

    bless($self, $type);
    return($self);
}

sub setValue($$$)
{
    my ($self, $key, $value)=@_;
    $self->{$key}=$value;
    if ( $key ne "option_postalcode" &&
	 $key ne "option_zipcode" &&
	 $key ne "option_provider" &&
	 $key ne "option_provider_desc" &&
	 $key ne "option_retry_limit" &&
	 $key ne "option_retry_delay" ) {
	die "attempt to set invalid key $key to $value";
    }
}

sub unsetValue($$$)
{
    my ($self, $key, $value)=@_;
    delete($self->{$key}) if ( defined($self->{$key}));
}

#
# get list of stations in display-name order
#
# sad but true, I can't figure a better way of
# doing this, but then again I don't care - jv
#
sub stationsInDisplayOrder($)
{
    my $self=shift;

    # create reverse hash with key/values swapped
    my @nums;
    foreach my $station (keys (%{$self->{channels}})) {
	my $key=0;
	if ( $station=~m/^\s*(\d+)/o ) {
	    $key=$1;
	}
	# seems very odd, but occasionally, you get two channels with the
	# same channel # on the dial
	if ( defined($nums[$key]) ) {
	    $nums[$key]="$nums[$key],$station";
	}
	else {
	    $nums[$key]="$station";
	}
    }
    
    my @ret;
    for (my $n=0; $n<scalar(@nums) ; $n++ ) {
	if ( defined($nums[$n]) ) {
	    push(@ret, sort (split(',', $nums[$n])))
	}
    }
    return(@ret);
}

sub haveAnyChannels($)
{
    my $self=shift;
    return(defined($self->{channels}));
}

sub stationRemove($$)
{
    my ($self, $station)=@_;
    delete($self->{channels}->{$station});
}

sub stationExists($$)
{
    my ($self, $station)=@_;
    return(defined($self->{channels}->{$station}));
}

sub setStationIncluded($$$)
{
    my ($self, $station, $in)=@_;
    $self->{channels}->{$station}->{in}=$in;
}

sub stationIncluded($$)
{
    my ($self, $station)=@_;
    return($self->{channels}->{$station}->{in});
}

sub stationIcon($$)
{
    my ($self, $station)=@_;
    return($self->{channels}->{$station}->{icon});
}

sub setStationIcon($$$)
{
    my ($self, $station, $icon)=@_;
    $self->{channels}->{$station}->{icon}=$icon;
}

sub setStationTransientFlag($$$$)
{
    my ($self, $station, $flag, $value)=@_;
    $self->{channels}->{$station}->{transient}->{$flag}=$value;
}

sub getStationTransientFlag($$$)
{
    my ($self, $station, $flag)=@_;
    return($self->{channels}->{$station}->{transient}->{$flag});
}

sub getStationChannelNumber($$)
{
    my ($self, $station)=@_;

    # commonly, stations are identified as '2 CFRN' so
    # we try and decode.
    if ( $station=~m/^\s*(\d+)\s*(.*)/o ) {
	return($1);
    }
    return(undef);
}

sub getStationChannelCallLetters($$)
{
    my ($self, $station)=@_;

    # commonly, stations are identified as '2 CFRN' so
    # we try and decode.
    if ( $station=~m/^\s*(\d+)\s*(.*)/o ) {
	return($2);
    }
    return(undef);
}

sub removeStationTransientFlag($$$)
{
    my ($self, $station, $flag)=@_;
    delete($self->{channels}->{$station}->{transient}->{$flag});
}

sub save($$)
{
    my ($self, $file)=@_;

    if ( main::mkpathtofile($file, 0) != 1 ) {
	print STDERR "mkdir failed on directory for $file:$!\n";
	return(-1);
    }

    open(FD, "> $file.new") || return(-1);
    print FD "# config file: tv_grab_na $VersionMajor.$VersionMinor\n";
    print FD "#\n";
    print FD "# this file is generated by running tv_grab_na --configure\n";
    print FD "# the only change you should make is prefixing 'channel:' lines\n";
    print FD "# with a 'not ' to signal that they should be ignored during the\n";
    print FD "# grab step\n";
    print FD "#\n";
    if ( defined($self->{option_retry_limit})) {
	print FD "retry limit: $self->{option_retry_limit}\n";
    }
    if ( defined($self->{option_retry_delay})) {
	print FD "retry delay: $self->{option_retry_delay}\n";
    }
    if ( defined($self->{option_postalcode})) {
	print FD "postal code: $self->{option_postalcode}\n";
    }
    if ( defined($self->{option_zipcode}) ) {
	print FD "zip code: $self->{option_zipcode}\n";
    }
    print FD "provider: $self->{option_provider} $self->{option_provider_desc}\n";

    foreach my $station ($self->stationsInDisplayOrder()) {
	if ( $self->stationIncluded($station) ) {
	    print FD "channel: $station\n"
	}
	else {
	    print FD "not channel: $station\n";
	}
    }
    if ( !close(FD) ) {
	unlink("$file.new");
	return(-1);
    }
    rename("$file.new", "$file") || return(-1);
    return(0);
}

sub load($$$)
{
    my ($self, $file, $debug)=@_;

    my $majorVersion;
    open(FD, "< $file") || return(-1);
    while (<FD>) {
	s/^\s+$//; s/\s+$//;

	if ( defined($majorVersion) ) {
	    # auto-upgrading from version 1 to 2
	    if ( $majorVersion == 1 ) {
		if ( m/^\#+channel:\s*(\S+)\s+\#\s+(.*)$/o ) {
		    $_="not channel: $2";
		}
		elsif ( m/^channel:\s*(\S+)\s+\#\s+(.*)$/o ) {
		    $_="channel: $2";
		}
	    }
	    # auto-upgrading from version 2 to 3
	    elsif ( $majorVersion == 2 ) {
		if ( m/^\#+channel:\s*(.+)$/o ) {
		    $_="not channel: $1";
		}
		elsif ( m/^provider:\s*(\S+)\s+\#\s+(.*)$/o ) {
		    $_="provider: $1 $2";
		}
	    }
	}

	# look for version number on first line only
	if ( $. == 1 ) {
	    if ( m/^\#?\s*config\s+file:\s*tv_grab_na (\d+)\.(\d+)/o ) {
		$majorVersion=$1;
		
		# identify and warn
		if ( $1 == 1 || $1 == 2 ) {
		    print STDERR "$file: format needs upgrading, re-run --configure\n";
		}
		elsif ( $1 != $VersionMajor || $2 > $VersionMinor ) {
		    print STDERR "$0: $file:$.: $1\.$2 is an unsupported version number\n";
		    close(FD) || warn "failed to close $file: $!";
		    return(-1);
		}
		next;
	    }
	    else {
		print STDERR "$file: didn't find version number, re-run --configure to fix\n";
		# and continue processing this line
	    }
	}

	if ( m/^retry limit:\s*(\d+)$/o ) {
	    $self->setValue("option_retry_limit", $1);
	}
	elsif ( m/^retry delay:\s*(\d+)$/o ) {
	    $self->setValue("option_retry_delay", $1);
	}
	elsif ( m/^postal code:\s*(\S+)$/o ) {
	    $self->setValue("option_postalcode", $1);
	}
	elsif ( m/^zip code:\s*(\S+)$/o ) {
	    $self->setValue("option_zipcode", $1);
	}
	elsif ( m/^provider:\s*(\S+)\s+(.*)$/o ) {
	    $self->setValue("option_provider", $1);
	    $self->setValue("option_provider_desc", $2);
	}
	elsif ( m/^not\s+channel:\s*(.+)$/o ) {
	    $self->setStationIncluded($1, 0);
	    #$self->setStationDescription($1, $1);
	}
	elsif ( m/^channel:\s*(.+)$/o ) {
	    $self->setStationIncluded($1, 1);
	    #$self->setStationDescription($1, $1);
	}
	elsif ( m/^#/o ) {
	    next;
	}
	elsif ( m/^\s*$/o ) {
	    # ignore empty lines
	}
	else {
	    warn "$0: $file:$.: bad line $_";
	    close(FD) || warn "failed to close $file: $!";
	    return(-1);
	}
	
    }
    close(FD) || return(-1);
    if ( defined($self->{option_postalcode}) && defined($self->{option_zipcode})) {
	print STDERR "$0: $file: corrupt, only one of postal or zip can be defined\n";
	return(-1);
    }
    return(0);
}

1;


package main;

sub writeOutChannels($$);

use strict;
#use diagnostics;
use Fcntl qw(:DEFAULT);
use XML::Writer;
use IO qw(File);
use Getopt::Long;
use File::Basename;
use Date::Manip;

use XMLTV::ZapListings;
use XMLTV::Ask;
use XMLTV::Config_file;
use XMLTV;
use XMLTV::Date;

my $xmltvSourceForgeURL="http://sourceforge.net/projects/xmltv";
my $xmltvFreshmeatRecord="http://freshmeat.net/projects-xml/xmltv/xmltv.xml";

my $now; # to be set later depending on --cache

our ($opt_debug, $opt_quiet, $opt_stats);

sub statusMessage($)
{
    my $msg = shift;
    die 'expected a message' if not defined $msg;
    if ( !$opt_quiet ) {
	say($msg);
    }
}

sub statsMessage($)
{
    my $msg = shift;
    die 'expected a message' if not defined $msg;
    if ( $opt_stats ) {
	say($msg);
    }
}

sub debugMessage($)
{
    my $msg = shift;
    die 'expected a message' if not defined $msg;
    if ( $opt_debug ) {
	print STDERR $msg;
    }
}

sub errorMessage($)
{
    my $msg = shift;
    die 'expected a message' if not defined $msg;
    say($msg);
}

sub mkpath($)
{
    my $path=shift;
    my @paths;

    if ( -d $path ) {
	return(1);
    }
    debugMessage("making path: $path..\n");
    while (length($path)!=0 && $path ne "." && $path ne "/" ) {
	last if ( -d $path );
	push(@paths, $path);
	$path=dirname($path);
    }

    foreach my $dir (reverse @paths) {
	if ( ! -d $dir ) {
	    mkdir($dir, 0775) || return(-1);
	}
	else {
	    debugMessage("$dir exists, not making\n");
	}
    }
    return(1);
}

sub mkpathtofile($)
{
    return(mkpath(dirname(shift)));
}

# Uses global $now.
sub writeListingsXMLHeader($)
{
    my $writer=shift;

    $writer->xmlDecl("ISO-8859-1");
    $writer->doctype('tv', undef, 'xmltv.dtd');
    die if not defined $now;
    my $now_formatted = Date::Manip::UnixDate($now,"%Y%m%d%H%M%S %z");
    die "cannot format $now" if not $now_formatted;
    $writer->startTag('tv',
		      date                  =>$now_formatted,
		      'source-info-url'     =>"http://www.zap2it.com",
		      'source-info-name'    =>"Zap2It",
		      #'source-data-url'     =>"",
		      'generator-info-name' =>"$VersionID",
		      'generator-info-url'  =>$xmltvSourceForgeURL);
}

sub writeListingsXMLFooter($)
{
    my $writer=shift;
    $writer->endTag('tv');
}

sub openOutputFileFD($$$$)
{
    my ($filename, $gzip_command, $zip_command, $bzip2_command)=@_;

    if ( $filename=~m/\.gz$/o ) {
        debugMessage("output filter: $gzip_command\n");
	return(new IO::File("| $gzip_command > $filename"));
    }
    elsif ( $filename=~m/\.zip$/o ) {
        debugMessage("output filter: $zip_command\n");
	return(new IO::File("| $zip_command > $filename"));
    }
    elsif ( $filename=~m/\.bz2$/o ) {
        debugMessage("output filter: $bzip2_command\n");
	return(new IO::File("| $bzip2_command > $filename"));
    }
    else {
	return(new IO::File("> $filename"));
    }
}

sub closeOutputFileFD($)
{
    $_[0]->close() if ( defined($_[0]) );
}

#
# How this grabber works:

# Step 1 - Configure
#  
#   The configure step is meant to be run interactively.
#   You can use 'tv_grab_na --configure --help' to see
#   how to run non-interactively, providing information on
#   the command line.

#   run 'tv_grab_na --configure'
#
#   Follow the prompts to provide the necessary information.
#
#   When finished, configure will create a file $HOME/.xmltv/$ConfigFileName_g
#   which contains the postal/zip code, the provider id
#   and a line for each channel that provider supplies.
#   This file is what you specify with the --config-file command
#   line option to "Step 2 Grabbing Data". See Step 2 for details
#   of how this file is interpreted.
#

sub ConfigureUsage($)
{
    no strict 'subs';
    no strict 'refs';
    my $stdout=shift;

    my $fp=STDERR;
    if ( $stdout ) {
	$fp=STDOUT;
    }
    print $fp "usage $0 --configure [options]\n";
    print $fp "where options are:\n";
    print $fp "   --help\n";
    print $fp "     print $fp configure help\n";
    print $fp "\n";
    print $fp "   --debug\n";
    print $fp "     turn on debugging\n";
    print $fp "\n";
    print $fp "   --postalcode XXXXXX\n";
    print $fp "     specify postal code, don't use with --zipcode\n";
    print $fp "\n";
    print $fp "   --zipcode YYYYYY\n";
    print $fp "     specify zip code, don't use with --postalcode\n";
    print $fp "\n";
    print $fp "   --provider ZZZZZZZ\n";
    print $fp "     specify provider id\n";
    print $fp "\n";
    print $fp "   --config-file <file>\n";
    print $fp "     use <file> as config file\n";
    print $fp "\n";
    print $fp "   --output <file>\n";
    print $fp "     redirect output to <file> rather than stdout. Currently only effects\n";
    print $fp "      --list-providers or --list-channels\n";
    print $fp "\n";
    print $fp "   --retry-limit <count>\n";
    print $fp "       upon possible www site failure, retry <count> times before hard failure.\n";
    print $fp "       (default 2)\n";
    print $fp "\n";
    print $fp "   --retry-delay <seconds>\n";
    print $fp "       number of seconds to sleep between retries (default 30).\n";
    print $fp "\n";
    print $fp "The following options are used to invoke --configure in a non-interactive mode\n";
    print $fp "(All three --auto* options should be specified)\n";
    print $fp "   --auto-fail-on-provider-changes <boolean>\n";
    print $fp "     true/false - exit 1 if provider has changed\n";
    print $fp "\n";
    print $fp "   --auto-new-channels [ignore|add]\n";
    print $fp "     ignore/add - ignore or automatically add new channels in schedule listing\n";
    print $fp "\n";
    print $fp "   --auto-missing-channels [ignore|remove]\n";
    print $fp "     ignore/remove - ignore or remove channels no longer in schedule listing\n";
    print $fp "\n";
    print $fp "The following options are available to support gui frontends who want to\n";
    print $fp "reproduce the configure process.\n";
    print $fp "\n";
    print $fp "   --list-providers\n";
    print $fp "     produce a list of providers and their descriptions.\n";
    print $fp "     requires one of --postalcode or --zipcode specified.\n";
    print $fp "     (use --output to redirect list to a file, rather than stdout)\n";
    print $fp "\n";
    print $fp "   --list-channels\n";
    print $fp "     produce a list of channels available.\n";
    print $fp "     requires --provider and one of --postalcode or --zipcode specifed.\n";
    print $fp "     (use --output to redirect list to a file, rather than stdout)\n";
    print $fp "\n";
    print $fp "   --release-check <true|false>\n";
    print $fp "     specify whether or not we should attempt to check for new releases\n";
    print $fp "     at $xmltvSourceForgeURL (default=true)\n";
    print $fp "     (output disabled if --quiet is used)\n";
    print $fp "\n";
    print $fp "If any neccessary options are given, interactive mode is enabled\n";
    print $fp "(ie no postal/zip code in config file and not on command line)\n";
}

# Step 2 - Grab
#   
#   The grab step uses the information collected during
#   configuration to get tv listings.
#  
#
#

sub Usage($)
{
    no strict 'subs';
    no strict 'refs';
    my $stdout=shift;

    my $fp=STDERR;
    if ( $stdout ) {
	$fp=STDOUT;
    }

    print $fp "usage:\n";
    print $fp "  $0 --configure [configure-options]\n";
    print $fp "  $0 [grab-options]\n";
    print $fp "command line options are:\n";
    print $fp "   --help\n";
    print $fp "     print this help\n";
    print $fp "     use --configure --help for configure help or\n";
    print $fp "     use --grab --help for grab help\n";
    print $fp "\n";
    print $fp "   --configure\n";
    print $fp "     run configuration step, see --configure --help for more info\n";
    print $fp "\n";
    print $fp "grab-options are:\n";
    print $fp "   --debug\n";
    print $fp "     turn on debugging\n";
    print $fp "\n";
    print $fp "   --debuglistings\n";
    print $fp "     add debugging material in output xml as comments\n";
    print $fp "\n";
    print $fp "   --quiet\n";
    print $fp "     disable all status messages (that normally appear on stderr).\n";
    print $fp "\n";
    print $fp "   --stats\n";
    print $fp "     force output of grab stats (stats output disabled in --quiet mode).\n";
    print $fp "\n";
    print $fp "   --config-file <file>\n";
    print $fp "     use <file> as config file\n";
    print $fp "\n";
    print $fp "   --output <file>\n";
    print $fp "     redirect listings to <file> instead of stdout. <file> may contain\n";
    print $fp "     L<date(1)> and keyword substitions like --listings.\n";
    print $fp "\n";
    print $fp "   --listings <file>\n";
    print $fp "     specify listings.xml filename(s) for channel & program info\n";
    print $fp "     <file> may contain Date::Manip::Unix substitutions\n";
    print $fp "     for instance, use --listings \"listings-%d%m%Y.xml\" to separate output\n";
    print $fp "     by day. Similarily, %postalcode, %zipcode, are also substituted from\n";
    print $fp "     settings in the current configuration. Channel-based keywords\n";
    print $fp "     %ChannelNumber and %ChannelCallLetters may be used, which\n";
    print $fp "     cause the filename to be re-evaluated between each channel\n";
    print $fp "     for each day instead of just when a new days listings is started.\n";
    print $fp "     if no --listings is specified stdout is used\n";
    print $fp "\n";
    print $fp "   --listings-overwrite <boolean>\n";
    print $fp "     should tv_grab_na overwrite existing --listings output files or skip the\n";
    print $fp "     ones that exist? (default=false - don't overwrite already grabbed\n";
    print $fp "     listings files)\n";
    print $fp "\n";
    print $fp "   --days n\n";
    print $fp "     specify number of days to include in output (default 7)\n";
    print $fp "\n";
    print $fp "   --offset n\n";
    print $fp "     specify number of days (in the future) to offset the\n";
    print $fp "     start of the listings (default 0)\n";
    print $fp "\n";
    print $fp "   --retry-limit <count>\n";
    print $fp "     upon possible www site failure, retry <count> times before hard failure.\n";
    print $fp "     (default 2)\n";
    print $fp "\n";
    print $fp "   --retry-delay <seconds>\n";
    print $fp "     number of seconds to sleep between retries (default 30).\n";
    print $fp "\n";
    print $fp "   --gzip-command <command>\n";
    print $fp "     specify the command line for .zip output files.(default is \"gzip\")\n";
    print $fp "\n";
    print $fp "   --zip-command <command>\n";
    print $fp "     specify the command line for .zip output files.(default is \"zip\")\n";
    print $fp "\n";
    print $fp "   --bzip2-command <command>\n";
    print $fp "     specify the command line for .bz2 output files.(default is \"bzip2\")\n";
    print $fp "\n";
    print $fp "   --release-check <true|false>\n";
    print $fp "     specify whether or not we should attempt to check for new releases\n";
    print $fp "     at $xmltvSourceForgeURL (default=true)\n";
    print $fp "     (output disabled if --quiet is used)\n";
    print $fp "\n";
}


# Check for configure mode or grab mode - they have different options.
# However, --list-channels and --list-providers imply configure mode,
# you don't need to give --configure with them explicitly.
our $opt_configure=0;
if ( @ARGV ) {
    foreach my $arg (@ARGV) {
	# Allow option abbreviation.
	if ( $arg=~m/^--configu/
	     or $arg=~m/^--list-c/ or $arg=~m/^--list-p/ ) {
	    $opt_configure=1;
	}
        if ( $arg=~m/^--debug$/o ) { $SIG{__WARN__} = sub { die $_[0] }; }
    }
}

if ( $opt_configure ) {
    our ($opt_help,
	 $opt_postalcode,
	 $opt_zipcode,
	 $opt_provider,
	 $opt_config_file,
	 $opt_output,
	 $opt_retry_limit,
	 $opt_retry_delay,
	 $opt_auto_fail_on_provider_changes,
	 $opt_auto_new_channels,
	 $opt_auto_missing_channels,
	 $opt_list_providers,
	 $opt_list_channels,
	 $opt_release_check,
	 $opt_cache,
	);
		
    $opt_auto_fail_on_provider_changes="false";
    $opt_release_check="true";

    if ( ! GetOptions(qw(configure
			 help
			 debug
			 quiet
			 release-check=s
			 config-file=s
			 output=s
			 postalcode=s
			 zipcode=s
			 provider=s
			 retry-limit=i
			 retry-delay=i
			 auto-fail-on-provider-changes=s
			 auto-new-channels=s
			 auto-missing-channels=s
			 list-providers
			 list-channels
                         cache:s
			))
	 or @ARGV # leftover arguments
       ) {
	print STDERR "use $0 --configure --help for usage\n";
	exit(1);
    }

    checkCache(); # make sure $opt_cache is a dir or undef

    $opt_release_check=($opt_release_check=~m;^t;i);

    $opt_quiet=(defined($opt_quiet));
    if ( !defined($opt_stats) ) {
	$opt_stats=!$opt_quiet;
    }
    else {
	$opt_stats=(defined($opt_stats));
    }

    if ( defined($opt_help) ) {
	ConfigureUsage(1);
	exit(0);
    }

    if ( defined($opt_list_providers) ) {
	if ( !defined($opt_postalcode) && !defined($opt_zipcode) ||
	     defined($opt_postalcode) && defined($opt_zipcode) ) {
	    errorMessage("$0: invalid usage, --list-providers requires ONE of --postalcode or --zipcode\n");
	    exit(1);
	}
	my $zl=new XMLTV::ZapListings('PostalCode'=>$opt_postalcode,
				      'ZipCode' => $opt_zipcode,
				      'Debug' => $opt_debug,
				      'CacheDir' => $opt_cache,
				     );
	if ( !defined($zl) ) {
	    exit(1);
	}
	
	my @providers=$zl->getProviderList();
	if ( ! @providers || !defined($providers[0]) ) {
	    exit(1);
	}
	# output list of providers to STDOUT
	my $outputFD;
	if ( defined($opt_output) ) {
	    if ( !open($outputFD, "> $opt_output") ) {
		errorMessage("$0: $opt_output: $!");
		exit(1);
	    }
	}
	else {
	    if ( !open($outputFD, ">&STDOUT") ) {
		errorMessage("$0: dup STDOUT failed: $!");
		exit(1);
	    }
	}
	foreach ( @providers ) {
	    my ($id, $desc) = ($_->{id}, $_->{description});
	    print $outputFD "$id:$desc\n";
	}
	close($outputFD);
	exit(0);
    }

    if ( defined($opt_list_channels) ) {
	if ( !defined($opt_provider) ) {
	    errorMessage("$0: invalid usage, --list-channels requires --provider specified\n");
	    exit(1);
	}
	if ( !defined($opt_postalcode) && !defined($opt_zipcode) ||
	     defined($opt_postalcode) && defined($opt_zipcode) ) {
	    errorMessage("$0: invalid usage, --list-channels requires ONE of --postalcode or --zipcode\n");
	    exit(1);
	}
	my $zl=new XMLTV::ZapListings('PostalCode'=>$opt_postalcode,
				      'ZipCode' => $opt_zipcode,
				      'Debug' => $opt_debug,
				      'CacheDir' => $opt_cache,
				     );
	if ( !defined($zl) ) {
	    exit(1);
	}
	my @channels=$zl->getChannelList($opt_provider);
	if ( ! @channels || !defined($channels[0]) ) {
	    exit(1);
	}
	my $config=new myConfig();
	foreach my $channel (@channels) {
	    my $station=$channel->{description};
	    $config->setStationIncluded($station, 1);
	    $config->setStationTransientFlag($station, 'found', 1);
	}

	my $outputFD;
	if ( defined($opt_output) ) {
	    if ( !open($outputFD, "> $opt_output") ) {
		errorMessage("$0: $opt_output: $!");
		exit(1);
	    }
	}
	else {
	    if ( !open($outputFD, ">&STDOUT") ) {
		errorMessage("$0: dup STDOUT failed: $!");
		exit(1);
	    }
	}

	# output list of channels to STDOUT or --output specified file
	my $writer = new XML::Writer(OUTPUT=>$outputFD,
				     DATA_MODE => 1, DATA_INDENT => 2 );
	die if not defined $now;
	writeListingsXMLHeader($writer);
	writeOutChannels($config, $writer);
	writeListingsXMLFooter($writer);
	$writer->end();
	close($outputFD);
	exit(0);
    }

    $opt_auto_fail_on_provider_changes=($opt_auto_fail_on_provider_changes=~m;^t;i);

    if ( defined($opt_auto_new_channels) ) {
	if ( $opt_auto_new_channels=~m;^ignore;i ) {
	    $opt_auto_new_channels="ignore";
	}
	elsif ( $opt_auto_new_channels=~m;^add;i ) {
	    $opt_auto_new_channels="add";
	}
	else {
	    errorMessage("$0: invalid argument to --auto_new_channels, must be 'ignore' or 'add'\n");
	    exit(1);
	}
    }

    if ( defined($opt_auto_missing_channels) ) {
	if ( $opt_auto_missing_channels=~m;^ignore;i ) {
	    $opt_auto_missing_channels="ignore";
	}
	elsif ( $opt_auto_missing_channels=~m;^remove;i ) {
	    $opt_auto_missing_channels="remove";
	}
	else {
	    errorMessage("$0: invalid argument to --auto_missing_channels, must be 'ignore' or 'remove'\n");
	    exit(1);
	}
    }

    my $config=new myConfig();

    # Pass the quiet flag, because we handle printing the message ourselves.
    my $configfile=XMLTV::Config_file::filename($opt_config_file, 'tv_grab_na', 1);
    #statusMessage("using config file $configfile\n");
    if ( -f $configfile && $config->load($configfile, $opt_debug) != 0 ) {
	errorMessage("$0: $configfile exists, but failed to read it\n");
	exit(1);
    }

    if ( defined($opt_postalcode) && defined($opt_zipcode) ) {
	errorMessage("$0: only one of --postalcode and --zipcode is allowed\n");
	exit(1);
    }

    # command line arguments
    if ( defined($opt_postalcode) ) {
	$config->setValue("option_postalcode", $opt_postalcode);
	$config->unsetValue("option_zipcode");
	$config->unsetValue("option_provider");
    }

    if ( defined($opt_zipcode) ) {
	$config->setValue("option_zipcode", $opt_zipcode);
	$config->unsetValue("option_postalcode");
	$config->unsetValue("option_provider");
    }

    if ( defined($opt_provider) ) {
	$config->setValue("option_provider", $opt_provider);
	$config->setValue("option_provider_desc", ""); # unknown so reset to ''
    }

    # sanity check
    if ( defined($config->{option_postalcode}) && defined($config->{option_zipcode})) {
	errorMessage("$0: only one of postal or zip code can be defined\n");
	exit(1);
    }

    if ( defined($opt_retry_limit) ) {
	$config->setValue("option_retry_limit", $opt_retry_limit);
    }

    if ( defined($opt_retry_delay) ) {
	$config->setValue("option_retry_delay", $opt_retry_delay);
    }

    #
    # Go interactive to collect what we don't have
    #

    my $msg="Welcome to XMLTV $XMLTV::VERSION ($VersionID) for Canada and US tv listings";
    statusMessage( <<END
$msg

Please report any problems, bugs or suggestions to:
	xmltv-users\@lists.sourceforge.net
	
For more information consult $xmltvSourceForgeURL

END
);

    if ( $opt_release_check ) {
	statusMessage("checking XMLTV release information..\n");

	my $retval=XMLTV::ZapListings::getCurrentReleaseInfo($xmltvFreshmeatRecord, $opt_debug);
	if ( !defined($retval) ) {
	    statusMessage( <<END
Warning: failed to get current release information from:
	$xmltvFreshmeatRecord
If this problem persists, look for a new XMLTV release.
END
);
	}
	else {
	    my %release=%{$retval};
	    if ( $release{VERSION} ne $XMLTV::VERSION ) {
		my $str="*** $release{NAME} $release{VERSION} available as of $release{DATESTRING} ***";
		statusMessage( "$str\nConsider upgrading." );
	    }
	    else {
		statusMessage("\nno worries, you're running the latest release\n");
	    }
	}
    }

    statusMessage("\nstarting manual configuration process..\n\n");

    while ( !defined($config->{option_retry_limit}) ) {
	my $res=ask('how many times do you want to retry on www site failures ? (default=2)');
	if ( !defined($res) || length($res) == 0 ) {
	    $res="2";
	}
	$res=~s/\s+//og if ( defined($res) );
	if ( defined($res) && length($res) ) {
	    $res=int($res);
	    if ( $res >= 0 ) {
		$config->setValue("option_retry_limit", $res);
		last;
	    }
	}
	errorMessage("$0: please specify an integer retry count\n\n");
    }

    while ( !defined($config->{option_retry_delay}) ) {
	my $res=ask('how many seconds do you want to between retries ? (default=30)');
	if ( !defined($res) || length($res) == 0 ) {
	    $res="30";
	}
	$res=~s/\s+//og if ( defined($res) );
	if ( defined($res) && length($res) ) {
	    $res=int($res);
	    if ( $res >= 0 ) {
		$config->setValue("option_retry_delay", $res);
		last;
	    }
	}
	errorMessage("$0: please specify an integer retry delay\n\n");
    }
    $opt_retry_limit=$config->{option_retry_limit};
    $opt_retry_delay=$config->{option_retry_delay};

    # if we have no postal code or zip code, the prompt for it
    if ( !defined($config->{option_postalcode}) && !defined($config->{option_zipcode})) {
	my $res=ask('what is your postal/zip code ?');
	$res=~s/\s+//og if ( defined($res) );
	if ( defined($res) && length($res) ) {
	   # $res=~tr/[a-z]/[A-Z]/;
	    if ( $res=~m/^[a-zA-Z]/o ) {
		$config->setValue("option_postalcode", $res);
	    }
	    else {
		$config->setValue("option_zipcode", $res);
	    }
	}
	else {
	    errorMessage("$0: failed to get postal/zip code\n");
	    exit(1);
	}
    }

    my $zl;

    # double check or get list of providers and give them the choice
    if ( 1 ) {

	my $code;
	$code=$config->{option_postalcode} if ( defined($config->{option_postalcode}) );
	$code=$config->{option_zipcode} if ( defined($config->{option_zipcode}) );

	statusMessage("\ngetting list of providers for postal/zip code $code, be patient..\n");

	my @providers;
	my $failed=0;
	for (my $retry=0 ; $retry<=$opt_retry_limit; $retry++ ) {
	    if ( $retry != 0 ) {
		errorMessage("failed $retry times, will retry after $opt_retry_delay seconds..\n");
		sleep($opt_retry_delay>2?$opt_retry_delay:2);
	    }
	    $zl=new XMLTV::ZapListings('PostalCode'=>$config->{option_postalcode}, 
				       'ZipCode' => $config->{option_zipcode},
				       'Debug' => $opt_debug,
				       'CacheDir' => $opt_cache);
	    if ( !defined($zl) ) {
		exit(1);
	    }
	    @providers=$zl->getProviderList();
	    if ( ! @providers || !defined($providers[0]) ) {
		$failed=1;
	    }
	    else {
		$failed=0;
		last;
	    }
	}
	if ( $failed ) {
	    #errorMessage("$0: failed to get list of providers for postal/zip code $code\n");
	    #errorMessage("   visit zap2it.com and try the postal/zip code for more information\n");
	    exit(1);
	}

	my $defaultProviderId;

	if ( $config->{option_provider} ) {
	    my $still_valid=0;
	    for my $p (@providers) {
		if ( $p->{id} eq $config->{option_provider} ) {
		    if ( $config->{option_provider_desc} ne $p->{description} ) {
			if ( $opt_auto_fail_on_provider_changes ) {
			    statusMessage("provider:\n\t".$config->{option_provider}." \# ".$config->{option_provider_desc}.
					  "\nhas new description (".$p->{description}."), exiting\n");
			    exit(1);
			}
			statusMessage("updating provider description to: $p->{description}\n");
			$config->{option_provider_desc}=$p->{description};
		    }
		    $still_valid=1;
		}
		if ( $p->{description} eq $config->{option_provider_desc} ) {
		    $defaultProviderId=$p->{id};
		}
	    }
	    # as a backup, check for a match based on the description, not the id
	    if ( $still_valid == 0 ) {
	        for my $p (@providers) {
		    if ( $config->{option_provider_desc} eq $p->{description} ) {
			if ( $p->{id} ne $config->{option_provider} ) {
			    if ( $opt_auto_fail_on_provider_changes ) {
				statusMessage("provider:\n\t".$config->{option_provider}." \# ".$config->{option_provider_desc}.
					      "\nhas new id (".$p->{id}."), exiting\n");
				exit(1);
			    }
			}
			statusMessage("updating provider id to: $p->{id}\n");
			$config->{option_provider}=$p->{id};
			$still_valid=1;
		    }
		}
	    }
	    if ( $still_valid == 0 ) {
		if ( $opt_auto_fail_on_provider_changes ) {
		    statusMessage("provider:\n\t".$config->{option_provider}." \# ".$config->{option_provider_desc}.
				  "\nis no longer valid, exiting\n");
		    foreach ( @providers ) {
			my ($id, $desc) = ($_->{id}, $_->{description});
			if ( $desc eq $config->{option_provider_desc} ) {
			    statusMessage("provider id has changed at zap2it, rerun configure interactively");
			    exit(1);
			}
		    }
		    statusMessage("provider string not found at zap2it, backup config file and rerun configure interactively");
		    exit(1);
		}
		statusMessage("provider:\n\t".$config->{option_provider}." \# ".$config->{option_provider_desc}.
			      "\nis no longer valid, choose a new one\n");
		delete($config->{option_provider});
		delete($config->{option_provider_desc});
	    }
	}

	while ( !$config->{option_provider} ) {
	    my @providersWithIds;

	    my (%descToId, %idToDesc);
	    die "no providers found" if not @providers;
	    foreach ( @providers ) {
		my ($id, $desc) = ($_->{id}, $_->{description});
		die if not defined $id;
		if ( not defined $desc ) {
		    statusMessage("warning: provider with id $id has no description\n");
		    next;
		}

		if ( exists $descToId{$desc} ) {
		    statusMessage("warning:two providers called $desc\n");
		}
		$descToId{$desc} = $id;
		if (exists $idToDesc{$id}) {
		    die "two providers with id $id\n";
		}
		$idToDesc{$id} = "$desc ($id)";
		push(@providersWithIds, "$desc ($id)");
	    }
	    if ( defined $defaultProviderId
		 and not exists $idToDesc{$defaultProviderId} ) {
		statusMessage("warning:cannot find default provider $defaultProviderId\n");
		undef $defaultProviderId;
	    }
	    if ( not defined $defaultProviderId ) {
		$providersWithIds[0]=~m/\(([^\)]+)\)$/o;
		$defaultProviderId =$1;
	    }

	    my $res = askQuestion("Choose a service provider: ",
				  $idToDesc{$defaultProviderId},
				  @providersWithIds);
	    die 'failed to choose service provider' if not defined $res;
	    $res=~s/\s+\(([^\)]+)\)$//o || die "how did you pick something not in the list ?";
	    my $id = $1; die if not defined $id;
	    $config->{option_provider}=$id;
	    $config->{option_provider_desc}=$res;
	    statusMessage("\nyou chose $id \# $res\n");
	}
    }

    # if we're in the configure step, lets refresh the list of channels
    # being careful to warn about additions and deletions
    
    if ( $config->haveAnyChannels() ) {
	statusMessage("\nchecking for changes to channel list, be patient..\n");
    }
    else {
	statusMessage("\ngetting channel list, be patient..\n");
    }

    my @channels;
    my $failed=0;
    for (my $retry=0 ; $retry<=$opt_retry_limit; $retry++ ) {
	if ( $retry != 0 ) {
	    errorMessage("failed $retry times, will retry after $opt_retry_delay seconds..\n");
	    sleep($opt_retry_delay>2?$opt_retry_delay:2);
	}

	@channels=$zl->getChannelList($config->{option_provider});
	statusMessage("got channel list\n");
	if ( ! @channels || !defined($channels[0]) ) {
	    $failed=1;
	}
	else {
	    $failed=0;
	    last;
	}
    }

    if ( $failed ) {
	#errorMessage("$0: failed to get list of channels for postal/zip code $config->{option_postalzipcode}\n");
	#errorMessage("   visit zap2it.com and try the postal/zip code for more information\n");
	exit(1);
    }

    my $channelsUpdated=0;

    # notify user about update channel ids and new channels
    my @toAsk;
    foreach my $channel (@channels) {
	my $station=$channel->{description};
	if ( $config->haveAnyChannels()
	     and $config->stationExists($station) ) {
	    $config->setStationTransientFlag($station, 'found', 1);
	}
	else {
	    push @toAsk, $station;
	}
    }
    
    # ask about each station, unless 'auto new channels' set
    my @r;
    if ( defined($opt_auto_new_channels) ) {
	if ( $opt_auto_new_channels eq "ignore" ) {
	    @r = map { 0 } @toAsk;
	    statusMessage("ignoring new channel $_\n") foreach @toAsk;
	}
	elsif ( $opt_auto_new_channels eq "add" ) {
	    @r = map { 1 } @toAsk;
	    statusMessage("adding new channel $_\n") foreach @toAsk;
	}
	else {
	    die "invalid auto_new_channels $opt_auto_new_channels\n";
	}
    }
    else {
	@r = askManyBooleanQuestions(1, map { "add channel $_ ?" } @toAsk);
    }

    # now use the Boolean answers in @r
    foreach (@toAsk) {
	my $w = shift @r;
	warn("cannot read input, stopping channel questions"), last
	  if not defined $w;
	$config->setStationIncluded($_, $w);
	$config->setStationTransientFlag($_, 'found', 1);
	#$config->setStationDescription($_, ...);
	$channelsUpdated++;
    }

    # warn about channel declarations we didn't find
    @toAsk = ();
    foreach my $station ($config->stationsInDisplayOrder()) {
	if ( defined($config->getStationTransientFlag($station, 'found')) ) {
	    $config->removeStationTransientFlag($station, 'found');
	}
	else {
	    push @toAsk, $station;
	}
    }

    @r = ();
    if ( defined($opt_auto_missing_channels) ) {
	if ( $opt_auto_missing_channels eq "ignore" ) {
	    @r = map { 0 } @toAsk;
	    statusMessage("ignoring missing channel $_...\n") foreach @toAsk;
	}
	elsif ( $opt_auto_missing_channels eq "remove" ) {
	    @r = map { 1 } @toAsk;
	    statusMessage("removing channel $_..\n") foreach @toAsk;
	}
	else {
	    die "invalid auto_missing_channels $opt_auto_missing_channels\n";
	}
    }
    else {
	@r = askManyBooleanQuestions(1, map { "remove no-longer available channel $_ ?" } @toAsk);
    }

    foreach my $station (@toAsk) {
	my $w = shift @r;
	warn("cannot read input, stopping channel questions"), last
	  if not defined $w;
	if ($w) {
	    #errorMessage("warning: didn't find channel id: $station \#".$config->stationDescription($station)."\n");
	    $config->stationRemove($station);
	    $channelsUpdated++;
	}
    }

    if ( $channelsUpdated == 0 ) {
	if ( $config->haveAnyChannels() ) {
	    statusMessage("\nchannel line-up hasn't changed\n");
	}
	else {
	    statusMessage("\nno channels added\n");
	}
    } 

    # write out config file
    statusMessage("\nupdating $configfile..\n");
    if ( $config->save($configfile) != 0 ) {
	errorMessage("$0: $configfile save failed\n");
	exit(1);

    }

    statusMessage("\nconfiguration step complete, let the games begin !\n");
    exit(0);
}

# in grabber mode - yeah !

our ($opt_help,
     $opt_config_file,
     $opt_programs,
     $opt_channels,
     $opt_listings,
     $opt_output,
     $opt_days,
     $opt_debuglistings,
     $opt_offset,
     $opt_retry_limit,
     $opt_retry_delay,
     $opt_gzip_command,
     $opt_zip_command,
     $opt_bzip2_command,
     $opt_listings_overwrite,
     $opt_release_check,
     $opt_cache,
    );

$opt_debuglistings=0;
$opt_gzip_command="gzip";
$opt_zip_command="zip";
$opt_bzip2_command="bzip2";
$opt_release_check="true";

if ( ! GetOptions(qw(help
		     debug
		     quiet
		     release-check=s
		     config-file=s
		     programs=s
		     channels=s
		     listings=s
		     output=s
		     days=i
		     debuglistings
		     offset=i
		     retry-limit=i
		     retry-delay=i
		     gzip-command=s
		     zip-command=s
		     bzip2-command=s
		     listings-overwrite=s
		     stats
                     cache:s
		    ))
     or @ARGV # leftover arguments
   ) {
    print STDERR "use $0 --help for usage\n";
    exit(1);
}

# Make sure $opt_cache is either an existing dir or undef.  Also set $now.
checkCache();

# TODO make the grabber independent of the local timezone.
#Date_Init('TZ=UTC');

# translate --quiet flag to a boolean
$opt_quiet=(defined($opt_quiet))?1:0;

# translate --release-check=[true|false]
$opt_release_check=($opt_release_check=~m;^t;i);

# stats are enabled if not specified or if no --quiet used
$opt_stats=(defined($opt_stats))?1:($opt_quiet==1)?0:1;

die 'number of days must not be negative'
  if (defined $opt_days && $opt_days < 0);

# throw error for invalid use of both --listings and --output
if ( defined($opt_output) && defined($opt_listings) ) {
    errorMessage("$0: only one of --listings or --output can be used at once\n");
    exit(1);
}

if ( defined($opt_output) && defined($opt_listings_overwrite) ) {
    errorMessage("$0: --listings-overwrite ignored when --output is used\n");
    $opt_listings_overwrite=undef;
}
if ( !defined($opt_listings_overwrite) ) {
    $opt_listings_overwrite="false";
}

if ( defined($opt_help) ) {
    Usage(1);
    exit(0);
}

#
# detect old style usage
#
if ( defined($opt_programs) || defined($opt_channels) ) {
    errorMessage("$0: new xmltv.dtd format, use --listings instead of\n");
    errorMessage("                --programs and/or --channels\n");
    exit(1);
}

# set defaults if they didn't appear on command line
$opt_days=7 if ( !defined($opt_days) );
$opt_offset=0 if ( !defined($opt_offset) );

if ( $opt_days < 0 || $opt_days > 14 ) {
    errorMessage("specified days must be between 1 and 14\n");
    exit(1);
}

STDOUT->autoflush(1);

my @FilesWeOpened_g;

my $failed=grab();

if ( $failed ) {
    if ( @FilesWeOpened_g ) {
	for my $file (@FilesWeOpened_g) {
	    if ( -f $file ) {
		errorMessage("removing $file after failure..\n");
		unlink($file) || warn("unable to remove tv_grab_na file: $file");
	    }
	}
    }
    exit(1);
}
exit(0);

# $opt_cache specifies a directory that ZapListings can use to cache
# some (not all) downloads.  This makes sure the dir exists.  Uses
# global $opt_cache and sets it to either a directory, or undef.
#
# This also sets the global $now to the date of the cache directory if
# it exists.
#
sub checkCache() {
    if ( defined($opt_cache) ) {
	if ( $opt_cache eq '' ) {
	    # default for historical reasons
	    $opt_cache="urldata";
	}
	if ( -d $opt_cache ) {
	    print STDERR "caching some downloads in $opt_cache\n";
	}
	else {
	    mkpath($opt_cache) || die "cannot mkdir $opt_cache: $!";
	    print STDERR "creating $opt_cache to cache some downloads\n";
	}
    }
    else {
	# default for historical reasons
	if ( -e "urldata" ) {
	    $opt_cache="urldata";
	    warn "$opt_cache exists, using it to cache some downloads\n";
	}
	# If it exists but is not a directory, we'll find out soon!
    }

    # Find current time, unless cache is used in which case travel
    # backwards in time.
    #
    # We don't currently call Date_Init() with a new timezone, but if
    # we did, it would be important for it to come after parsing
    # 'now', and perhaps for 'now' to be parsed with
    # parse_local_date().  (Bug in Date::Manip.)
    #
    if ( defined $opt_cache ) {
	my @stat = stat($opt_cache);
	die "cannot stat $opt_cache: $!" if not @stat;
	my $ctime = $stat[10];
	$now = parse_date("epoch $ctime");
	warn 'using date ' . UnixDate($now, '%q') . " from $opt_cache\n";
    }
    else {
	$now = parse_date("now");
    }
}

sub grab
{
    # initalize global XML::Writer if we're writting listings to stdout
    # or if all listings are to being output'd to a single file.
    # The later allows us to use --listings tv.xml with more than one days listings.
    my $writer_g;

    # output_g is only used if we are writting all Listings to the same file.
    my $output_g;

    my $config=new myConfig();

    # Pass the quiet flag, because we handle printing the message ourselves.
    my $configfile=XMLTV::Config_file::filename($opt_config_file, 'tv_grab_na', 1);
    statusMessage("using config file $configfile\n");
    if ( not -f $configfile ) {
	errorMessage("$0: Config file $configfile does not exist, run me with --configure\n");
	exit(1);
    }
    if ( $config->load($configfile, $opt_debug) != 0 ) {
	errorMessage("$0: Failed to read $configfile\n");
	return(1);
    }

    if ( !defined($opt_retry_limit) ) {
	if ( !defined($config->{option_retry_limit}) ) {
	    errorMessage("no retry limit set in configuration, re-run --configure\n");
	    $opt_retry_limit=2;
	}
	else {
	    $opt_retry_limit=$config->{option_retry_limit};
	}
    }

    if ( !defined($opt_retry_delay) ) {
	if ( !defined($config->{option_retry_delay}) ) {
	    errorMessage("no retry delay set in configuration, re-run --configure\n");
	    $opt_retry_delay=30;
	}
	else {
	    $opt_retry_delay=$config->{option_retry_delay};
	}
    }

    # check provider information, usually fast anyway.
    #
    my $zl;
    if ( 1 ) {
	my $code;
	$code=$config->{option_postalcode} if ( defined($config->{option_postalcode}) );
	$code=$config->{option_zipcode} if ( defined($config->{option_zipcode}) );

	statusMessage("\nchecking provider information for postal/zip code $code, be patient..\n");

	my @providers;
	my $failed=0;
	for (my $retry=0 ; $retry<=$opt_retry_limit; $retry++ ) {
	    if ( $retry != 0 ) {
		errorMessage("failed $retry times, will retry after $opt_retry_delay seconds..\n");
		sleep($opt_retry_delay>2?$opt_retry_delay:2);
	    }
	    $zl=new XMLTV::ZapListings('PostalCode'=>$config->{option_postalcode}, 
				       'ZipCode' => $config->{option_zipcode},
				       'Debug' => $opt_debug,
				       'DebugListings'=>$opt_debuglistings,
				       'CacheDir' => $opt_cache);
	    if ( !defined($zl) ) {
		exit(1);
	    }
	    @providers=$zl->getProviderList();
	    if ( ! @providers || !defined($providers[0]) ) {
		$failed=1;
	    }
	    else {
		$failed=0;
		last;
	    }
	}
	if ( $failed ) {
	    #errorMessage("$0: failed to get list of providers for postal/zip code $code\n");
	    #errorMessage("   visit zap2it.com and try the postal/zip code for more information\n");
	    exit(1);
	}

	my $still_valid=0;
	for my $p (@providers) {
	    if ( $p->{id} eq $config->{option_provider} ) {
		if ( $config->{option_provider_desc} ne $p->{description} ) {
		    statusMessage("recoverable change (provider description changed to ($p->{description})), think about re-running --configure\n");
		    $config->{option_provider_desc}=$p->{description};
		}
		$still_valid=1;
	    }
	}
	if ( $still_valid == 0 ) {
	    for my $p (@providers) {
		if ( $config->{option_provider_desc} eq $p->{description} ) {
		    if ( $p->{id} ne $config->{option_provider} ) {
			statusMessage("recoverable change (provider id changed to ($p->{id})), think about re-running --configure\n");
			$config->{option_provider}=$p->{id};
		    }
		    $still_valid=1;
		}
	    }
	}
	if ( $still_valid == 0 ) {
	    errorMessage("configured provider no longer valid (for postal/zip code $code), re-run --configure\n");
	    # return failed
	    return(1);
	}
    }

    # collect information about channels.
    if ( 1 ) {
	statusMessage("double checking channel information, be patient..\n");

	my @channels;
	my $failed=0;
	for (my $retry=0 ; $retry<=$opt_retry_limit; $retry++ ) {
	    if ( $retry != 0 ) {
		errorMessage("failed $retry times, will retry after $opt_retry_delay seconds..\n");
		sleep($opt_retry_delay>2?$opt_retry_delay:2);
	    }

	    @channels=$zl->getChannelList($config->{option_provider});
	    if ( ! @channels || !defined($channels[0]) ) {
		$failed++;
		#errorMessage("$0: failed to get list of channels for postal/zip code $config->{option_postalzipcode}\n");
		#errorMessage("   visit zap2it.com and try the postal/zip code for more information\n");
		#exit(1);
	    }
	    else {
		$failed=0;
		last;
	    }
	}
	if ( $failed ) {
	    exit(1);
	}
	
	my $channelsUpdated=0;

	# notify user about update channel ids and new channels
	foreach my $channel (@channels) {
	    my $station=$channel->{description};
	    if ( $config->haveAnyChannels() ) {
		if ( $config->stationExists($station) ) {
		    $config->setStationTransientFlag($station, 'found', 1);

		    # save zap2it channel id for grabbing url usage
		    $config->setStationTransientFlag($station, 'zap2it-id', $channel->{stationid});
		    
		    if ( defined($channel->{icon}) ) {
			$config->setStationIcon($station, $channel->{icon});
		    }
		}
		else {
		    errorMessage("noticed new station available ($station), re-run --configure\n");
		}
	    }
	    else {
		errorMessage("noticed new station available ($station), re-run --configure\n");
	    }
	}

	# warn about channel declarations we didn't find
	foreach my $station ($config->stationsInDisplayOrder()) {
	    if ( defined($config->getStationTransientFlag($station, 'found')) ) {
		$config->removeStationTransientFlag($station, 'found');
	    }
	    else {
		errorMessage("noticed station unavailable ($station), re-run --configure\n");
		$config->setStationTransientFlag($station, 'notavailable', 1);
	    }
	}

	if ( $channelsUpdated++ ) {
	    errorMessage("some channel information is out of date, re-run --configure\n");
	}
    }

    my $stats;

    $stats->{num_channels}=0;
    $stats->{num_programs}=0;
    $stats->{num_days}=0;
    $stats->{num_readScheduleFailed}=0;
    $stats->{num_alreadyHad}=0;

    # start time only includes programming grabs, no channel detail grab
    my $startTime=time();
    my $listingsDefinition;
    my $overwrite;

    # implement --output as synonym for --listings
    if ( defined($opt_output) ) {
	$listingsDefinition=$opt_output;
	undef($opt_output);
	$overwrite=1;
    }
    elsif ( defined($opt_listings) ) {
	$listingsDefinition=$opt_listings;
	$overwrite=($opt_listings_overwrite eq "true");
    }
    else {
	$listingsDefinition=undef; # stdout
	$overwrite=0;
    }

    if ( defined($listingsDefinition) ) {
	# do %postalcode and %zipcode substitutions now since they can't change
	# if they don't appear, we remove them
	if ( defined($config->{option_postalcode}) ) {
	    $listingsDefinition=~s/%(postal|zip)code/$config->{option_postalcode}/og;
	}
	elsif ( defined($config->{option_zipcode}) ) {
	    $listingsDefinition=~s/%(postal|zip)code/$config->{option_zipcode}/og;
	}
	else {
	    $listingsDefinition=~s/%(postal|zip)code//og;
	}
	$listingsDefinition=~s/%(Channel)/%%$1/og;
    }

    my $skipAllGrabs;
    if ( !defined($listingsDefinition) ) {
	statusMessage("writing listings to stdout\n");
	$writer_g = new XML::Writer(DATA_MODE => 1, DATA_INDENT => 2 );
	writeListingsXMLHeader($writer_g);
	$stats->{num_channels}=writeOutChannels($config, $writer_g);
	$skipAllGrabs=0;
    }
    elsif ( UnixDate("now","$listingsDefinition") eq $listingsDefinition ) {
	if ( !$overwrite && -f $listingsDefinition ) {
	    statusMessage("listings in $listingsDefinition exist, skipping\n");
	    $stats->{num_alreadyHad}++;
	    $skipAllGrabs=1;
	}
	else {
	    mkpathtofile($listingsDefinition) || die "mkdir $listingsDefinition:$!";
	    statusMessage("writing listings to $listingsDefinition\n");
	    $output_g = openOutputFileFD($listingsDefinition,
					 $opt_gzip_command,
					 $opt_zip_command,
					 $opt_bzip2_command) || die "$listingsDefinition: $!";
	    push(@FilesWeOpened_g, $listingsDefinition);
	    $writer_g = new XML::Writer(OUTPUT=>$output_g,
					DATA_MODE => 1, DATA_INDENT => 2 );
	    writeListingsXMLHeader($writer_g);
	    $stats->{num_channels}=writeOutChannels($config, $writer_g);
	    $skipAllGrabs=0;
	}
    }

    if ( !$skipAllGrabs ) {
 	my $failedCount=0;
	my ($y,$m,$d,$h,$mn,$s)=Date::Manip::Date_Split($now);
	my $startNDay=Date_DayOfYear($m,$d,$y);
	my $tz=Date_TimeZone();
	
	my $failHardOnReadScheduleFailure=0;
	
	#
	# So that the output a day at a time, this allows for separate files per day
	#
	my $year=$y;
	my $runNDayOfYear=$startNDay + $opt_offset;
	for (my $nday=0; $nday<$opt_days ; $nday++) {
	    my $writer;
	    my $output;

	    # handle cross-year listings
	    while ( $runNDayOfYear+$nday > Date_DaysInYear($year) ) {
		$runNDayOfYear-=Date_DaysInYear($year);
		$year++;
	    }
	
	    $stats->{num_days}++;;

	    my $dateStr=createDateString(0, $runNDayOfYear+$nday, $year, 0, $tz);

	    my $dayFilename;
	    my $skipGrabThisFileExists;
	    if ( defined($writer_g) ) {
		$writer=$writer_g;
		$skipGrabThisFileExists=0;
	    }
	    else {
		$dayFilename=UnixDate($dateStr, "$listingsDefinition");
		if ( $listingsDefinition eq $dayFilename ) {
		    die "This case should have been caught before here";
		}
		if ( !$overwrite && -f $dayFilename ) {
		    statusMessage("listings in $dayFilename exist, skipping\n");
		    $stats->{num_alreadyHad}++;
		    $skipGrabThisFileExists=1;
		}
		else {
		    if ( !($dayFilename=~m/%Channel/o) ) {
			mkpathtofile($dayFilename) || die "mkdir $dayFilename:$!";
			statusMessage("writing listings to $dayFilename\n");
			$output = openOutputFileFD($dayFilename,
						   $opt_gzip_command,
						   $opt_zip_command,
						   $opt_bzip2_command) || die "$dayFilename: $!";
			$writer = new XML::Writer(OUTPUT=>$output,
						  DATA_MODE => 1, DATA_INDENT => 2 );
			writeListingsXMLHeader($writer);
			$stats->{num_channels}=writeOutChannels($config, $writer);
		    }
		    $skipGrabThisFileExists=0;
		}
	    }
	    
	    if ( !$skipGrabThisFileExists ) {
		my $failEntireDay=0;

		foreach my $station ($config->stationsInDisplayOrder()) {
		    
		    next if ( !$config->stationIncluded($station) );
		    
		    if ( defined($config->getStationTransientFlag($station, 'notavailable')) ) {
			statusMessage("skipping unavailable channel $station\n");
			next;
		    }
		    
		    if ( !defined($config->getStationTransientFlag($station, 'zap2it-id')) ) {
			warn "ignoring channel without zap2it channel id $station\n";
			next;
		    }
		    
		    my ($Year,$month,$day,$hr,$min,$sec)=Date::Manip::Date_NthDayOfYear($year, $runNDayOfYear+$nday);
		    
		    for (my $retry=0 ; $retry<=$opt_retry_limit; $retry++ ) {
			if ( $retry != 0 ) {
			    errorMessage("failed $retry times, will retry after $opt_retry_delay seconds..\n");
			    sleep($opt_retry_delay>2?$opt_retry_delay:2);
			}
			my $ret=$zl->readSchedule($config->getStationTransientFlag($station, 'zap2it-id'),
						  $station,
						  $day, $month, $Year);
			if ( $ret == -1) {
			    errorMessage("failed to read schedule for $Year-$month-$day for station $station\n");
			    # fail hard on first failure
			    $failedCount++;
			}
			elsif ( $ret == -2 ) {
			    errorMessage("unretry-able error reading schedule for $Year-$month-$day for station $station\n");
			    # fail hard on first failure
			    $failedCount++;
			    last;
			}
			else {
			    $failedCount=0;
			    last;
			}
		    }
		    
		    if ( $failedCount ) {
			# hard failure means fail the entire grab when any channel's
			# schedule fails to come
			if ( $failHardOnReadScheduleFailure ) {
			    if ( !defined($writer_g) || $writer != $writer_g ) {
				writeListingsXMLFooter($writer);
				$writer->end();
				closeOutputFileFD($output);
			    }
			    if ( defined($writer_g) ) {
				writeListingsXMLFooter($writer_g);
				$writer_g->end();
				closeOutputFileFD($output_g) if ( defined($output_g) );
			    }
			    if ( defined($dayFilename) ) {
				unlink($dayFilename) || warn("unable to remove tv_grab_na file: $dayFilename");
			    }
			    return($failedCount);
			}
			else {
			    # if we're writting to files, one per day, fail entire
			    # day (the file) if any channel fails
			    if ( !defined($writer_g) || $writer != $writer_g ) {
				$failEntireDay=1;
				last; # break early
			    }
			    else {
				# writting to stdout, so just track these
				$stats->{num_readScheduleFailed}++;
			    }
			}
		    }
		    else {
			# set time zone for date conversions
			#Date::Manip::Date_SetConfigVariable('TZ', $zl->{TimeZone});
			#Date::Manip::Date_Init();
			
			# if no writer exists, then we're suppose to generate one after every successful channel
			# grab. This should only occur if some %Channel* attribute exists in the filename.
			if ( !defined($writer) ) {
			    my $filename=$dayFilename;
			    
			    my $number=$config->getStationChannelNumber($station);
			    my $callletters=$config->getStationChannelCallLetters($station);

			    if ( $filename=~m/%ChannelNumber/o ) {
				if ( !defined($number) ) {
				    errorMessage("you've used \%ChannelNumber in your output filename, but Channel \"$station\" failed to provide one");
				    errorMessage("please forward details of this problem to xmltv-devel\@lists.sf.net");
				    last; # break early
				}
				$filename=~s/%ChannelNumber/$number/og;
			    }
			    if ( $filename=~m/%ChannelCallLetters/o ) {
				if ( !defined($callletters) ) {
				    errorMessage("you've used \%ChannelCallLetters in your output filename, but Channel \"$station\" failed to provide one");
				    errorMessage("please forward details of this problem to xmltv-devel\@lists.sf.net");
				    last; # break early
				}
				$filename=~s/%ChannelCallLetters/$callletters/og;
			    }

			    if ( $filename eq $dayFilename ) {
				die "how did we get here with dayFilename=$dayFilename ?";
			    }

			    if ( !$overwrite && -f $filename ) {
				statusMessage("listings in $filename exist, skipping\n");
				$stats->{num_alreadyHad}++;
			    }
			    else {
				mkpathtofile($filename) || die "mkdir $filename:$!";
				statusMessage("writing listings to $filename\n");
				my $o = openOutputFileFD($filename,
							 $opt_gzip_command,
							 $opt_zip_command,
							 $opt_bzip2_command) || die "$dayFilename: $!";
				my $w = new XML::Writer(OUTPUT=>$o,
							DATA_MODE => 1, DATA_INDENT => 2 );
				writeListingsXMLHeader($w);
				$stats->{num_channels}+=writeOutChannel($config, $w, $station);
				$stats->{num_programs}+=writeOutPrograms($config, $zl, $runNDayOfYear+$nday, $Year, $tz, $w, $station);
				writeListingsXMLFooter($w);
				$w->end();
				closeOutputFileFD($o);
			    }
			}
			else {
			    $stats->{num_programs}+=writeOutPrograms($config, $zl, $runNDayOfYear+$nday, $Year, $tz, $writer, $station);
			}
		    }
		}

		if ( !defined($writer_g) || $writer != $writer_g ) {
		    if ( defined($writer) ) {
			writeListingsXMLFooter($writer);
			$writer->end();
			closeOutputFileFD($output);
		    }
		}
		if ( $failEntireDay ) {
		    if ( defined($dayFilename) ) {
			errorMessage("channel schedule failed, so we're removing $dayFilename\n");
			unlink($dayFilename) || warn("unable to remove tv_grab_na file: $dayFilename");
		    }
		}
		else {
		    push(@FilesWeOpened_g, $dayFilename) if ( defined($dayFilename) );
		}
	    }
	}
    }

    if ( defined($writer_g) ) {
	writeListingsXMLFooter($writer_g);
	$writer_g->end();
	closeOutputFileFD($output_g) if ( defined($output_g) );
    }

    if ( $opt_stats ) {
	my $endTime=time();

	$stats->{calcProgramsPerSecond}=($endTime!=$startTime &&
					 $stats->{num_programs} != 0)?
					     $stats->{num_programs}/($endTime-$startTime): 0;
	
	$stats->{calcSecondsPerlWWWPage}=($endTime!=$startTime &&
					  $stats->{num_days}*$stats->{num_channels}!=0)?
					      ($endTime-$startTime)/($stats->{num_days}*$stats->{num_channels}): 0;
	
	statsMessage(sprintf("Grabbed %d programs on %d channels over %d day(s) in %d seconds\n",
			     $stats->{num_programs},
			     $stats->{num_channels},
			     $stats->{num_days},
			     $endTime-$startTime));
	
	statsMessage(sprintf("  not too bad, that's %.2f programs/sec and %.2f seconds/www page\n",
			     $stats->{calcProgramsPerSecond}, $stats->{calcSecondsPerlWWWPage}));
	
	if ( $stats->{num_readScheduleFailed} != 0 ) {
	    statsMessage(sprintf("  does not include %d failed channel schedules\n",
				 $stats->{num_readScheduleFailed}));
	}
	if ( $stats->{num_alreadyHad} != 0 ) {
	    statsMessage(sprintf("  does not include %d output file(s) that already existed\n",
				 $stats->{num_alreadyHad}));
	}
    }
    
    if ( $opt_release_check ) {
	my $retval=XMLTV::ZapListings::getCurrentReleaseInfo($xmltvFreshmeatRecord, $opt_debug);
	if ( !defined($retval) ) {
	    statusMessage("\nWarning: failed to get current release information from:\n");
	    statusMessage("   $xmltvFreshmeatRecord\n");
	    statusMessage("If this problem persists, look for a new XMLTV release\n\n");
	}
	else {
	    my %release=%{$retval};
	    if ( $release{VERSION} ne $XMLTV::VERSION ) {
		my $str="*** newer release ($release{NAME} $release{VERSION}) was available $release{DATESTRING} ***";
		statusMessage("*" x length($str)."\n$str\n"."*" x length($str)."\n");
	    }
	    else {
		statusMessage("Nice going, you're running the latest release of XMLTV\n");
	    }
	}
    }
    return(0);
}

# create a conversion string
sub createDateString($$$$$)
{
    my ($minuteOfDay, $dayOfYear, $year, $additionalMin, $time_zone)=@_;
    
    if ( $additionalMin != 0 ) {
	$minuteOfDay+=$additionalMin;

	# deal with case where additional minutes pushes us over end of day
	if ( $minuteOfDay > 24*60 ) {
	    $minuteOfDay-=24*60;
	    $dayOfYear++;

	    # check and deal with case where this pushes us past end of year
	    my $isleap=Date_LeapYear($year);
	    if ($dayOfYear >= ($isleap ? 367 : 366)) {
		$year++;
		$dayOfYear-=($isleap ? 367 : 366);
	    }
	}
    }

    # account for end of year boundaries
    while ( $dayOfYear > Date_DaysInYear($year) ) {
	$dayOfYear-=Date_DaysInYear($year);
	$year++;
    }

    # calculate year,month and day from nth day of year info
    my ($pYEAR,$pMONTH,$pDAY,$pHR,$pMIN,$pSEC)=Date::Manip::Date_NthDayOfYear($year, $dayOfYear);

    # set HR and MIN to what they should really be
    $pHR=int($minuteOfDay/60);
    $pMIN=$minuteOfDay-($pHR*60);

    return(sprintf("%4d%02d%02d%02d%02d00 %s", $pYEAR, $pMONTH, $pDAY, $pHR, $pMIN, $time_zone));
}

sub writeOutPrograms($$$$$$$)
{
    my ($config, $zl, $dayOfYear, $year, $mytz, $writer, $station)=@_;
    my $IncludePartialPrograms=0;

    my $number=$config->getStationChannelNumber($station);
    my $letters=$config->getStationChannelCallLetters($station);
    my $channel;

    if ( defined($number) && defined($letters) ) {
	$channel="C".$number.lc($letters).".zap2it.com";
    }
    else {
	$channel="$station";
    }

    my @programs=$zl->getPrograms();

    for my $progTop (@programs) {
	my $prog=$progTop;

	if ( $opt_debug ) {
	    debugMessage("before write: prog:".XMLTV::ZapListings::dumpMe($prog)."\n");
	}

	$prog->{start}=createDateString(($prog->{start_hour}*60+$prog->{start_min}), $dayOfYear, $year, 0, $mytz);
	delete($prog->{start_hour});
	delete($prog->{start_min});
	if ( defined($prog->{end_hour}) ) {
	   if ( $prog->{end_hour} >= 24 ) {
	       $prog->{end}=createDateString((($prog->{end_hour}-24)*60+$prog->{end_min}), $dayOfYear+1, $year, 0, $mytz);
	   }
	   else {
	       $prog->{end}=createDateString(($prog->{end_hour}*60+$prog->{end_min}), $dayOfYear, $year, 0, $mytz);
	   }
	   delete($prog->{end_hour});
	   delete($prog->{end_min});
	}

	my $title=$prog->{title};
	delete($prog->{title});
	    
	#debugMessage("storing $title..\n");

	if ( defined($prog->{contFromPreviousListing}) ) {
	    if ( !$IncludePartialPrograms ) {
		#errorMessage("warning: not including program $prog->{title} which starts previous to listing\n");
		next;
	    }
	    $title="(<-cont) $title";
	    delete($prog->{contFromPreviousListing});
	}
	if ( defined($prog->{contToNextListing}) ) {
	    if ( !$IncludePartialPrograms ) {
		#errorMessage("warning: not including program $prog->{title} which ends past listing boundaries\n");
		next;
	    }
	    $title="$title (cont->)";
	    delete($prog->{contToNextListing});
	}
	
	if ( defined($prog->{end}) ) {
	   $writer->startTag('programme', start=> $prog->{start}, 
			     stop => $prog->{end}, channel=> "$channel");
	   delete($prog->{start});
	   delete($prog->{end});
	}
	else {
	   $writer->startTag('programme', start=> $prog->{start}, 
			     channel=> "$channel");
	   delete($prog->{start});
	   #delete($prog->{end});
	}

	if ( defined($prog->{precomment}) ) {
	    # -- in comments causes carp() calls in Writer.pm
	   $prog->{precomment}=~s/\-\-/\=\=/og;

	   # imbed in programme output a comment
	   # this is here for debugging listings and their results
	   $writer->comment("\n   ".$prog->{precomment});
	   delete($prog->{precomment});
	}
	$writer->dataElement('title', $title);
	if ( defined($prog->{subtitle}) ) {
	    $writer->dataElement('sub-title', $prog->{subtitle});
	    delete($prog->{subtitle});
	}

	if ( defined($prog->{qualifiers}) ) {
	    for my $qf ('Live', 'Taped', 'If Necessary', 'Subject to Blackout', 'HDTV') {
		if ( defined($prog->{qualifiers}->{$qf}) ) {
		    # nowhere in xmltv.dtd to put this
		    if ( defined($prog->{desc}) ) {
			if ( $prog->{desc}=~m/\)$/o ) {
			    $prog->{desc}.="($qf)" ;
			}
			else {
			    $prog->{desc}.=" ($qf)" ;
			}
		    }
		    delete($prog->{qualifiers}->{$qf});
		}
	    }
	}

	if ( defined($prog->{desc}) ) {
	    $writer->dataElement('desc', $prog->{desc});
	    delete($prog->{desc});
	}
		
	if ( defined($prog->{director}) || defined($prog->{actors}) ) {
	    $writer->startTag('credits');
	    $writer->dataElement('director', $prog->{director}) if ( defined($prog->{director}) );
	    if ( defined($prog->{actors}) ) {
		foreach my $actor (@{$prog->{actors}}) {
		    $writer->dataElement('actor', $actor);
		}
	    }
	    $writer->endTag('credits');
	    delete($prog->{director}) if ( defined($prog->{director}) );
	    delete($prog->{actors}) if ( defined($prog->{actors}) );
	}
	if ( defined($prog->{year}) ) {
	    $writer->dataElement('date', $prog->{year});
	    delete($prog->{year});
	}
	
	if ( defined($prog->{category}) ) {
	    foreach my $cat(@{$prog->{category}}) {
		$writer->dataElement('category', $cat);
	    }
	    delete($prog->{category});
	}

	# hunt for things in the "qualifiers bucket" where alot of things fall.
	if ( defined($prog->{qualifiers}) ) {
	    if ( defined($prog->{qualifiers}->{Language}) ) {
		if ( defined($prog->{qualifiers}->{Dubbed}) ) {
		    $writer->dataElement('orig-language', $prog->{qualifiers}->{Language});
		    $writer->dataElement('language', $prog->{qualifiers}->{Dubbed});
		    delete($prog->{qualifiers}->{Dubbed});
		}
		else {
		    $writer->dataElement('language', $prog->{qualifiers}->{Language});
		}
		delete($prog->{qualifiers}->{Language});
	    }
	    
	    for ( my $pi = delete($prog->{qualifiers}->{PartInfo}) ) {
		if ( defined ) {
		    if ( /^Part (\d+) of (\d+)$/
			 and $1 > 0
			 and $1 <= $2 ) {
			my ($from_zero, $out_of) = ($1 - 1, $2);
			$writer->dataElement('episode-num',
					     " .  . $from_zero/$out_of",
					     system => 'xmltv_ns');
		    }
		    else {
			warn "discarding unrecognized part information: $_\n";
		    }
		}
	    }

	    if ( defined($prog->{qualifiers}->{BlackAndWhite}) ||
		 defined($prog->{qualifiers}->{VideoAspect}))  {
		$writer->startTag('video');
		if ( defined($prog->{qualifiers}->{VideoAspect}) ) {
		    $writer->dataElement('aspect', $prog->{qualifiers}->{VideoAspect});
		    delete($prog->{qualifiers}->{VideoAspect});
		}
		if ( defined($prog->{qualifiers}->{BlackAndWhite}) ) {
		    $writer->dataElement('colour', 'no');
		    delete($prog->{qualifiers}->{BlackAndWhite});
		}
		$writer->endTag('video');
	    }
	    if ( defined($prog->{qualifiers}->{InStereo}) ) {
		$writer->startTag('audio');
		# The 'stereo' element requires some text inside it,
		# so you have to say <stereo>stereo</stereo>.  :-P.
		$writer->dataElement('stereo', 'stereo');
		$writer->endTag('audio');
		delete($prog->{qualifiers}->{InStereo});
	    }
	    if ( defined($prog->{qualifiers}->{PreviouslyShown}) ) {
		# Write as <previously-shown /> to indicate there is
		# no textual content, not even the empty string :-).
		$writer->emptyTag('previously-shown');
		delete($prog->{qualifiers}->{PreviouslyShown});
	    }
	    if ( defined($prog->{qualifiers}->{PremiereShowing}) ) {
		$writer->dataElement('premiere',$prog->{qualifiers}->{PremiereShowing});
		delete($prog->{qualifiers}->{PremiereShowing});
	    }
	    if ( defined($prog->{qualifiers}->{LastShowing}) ) {
		$writer->dataElement('last-chance',$prog->{qualifiers}->{LastShowing});
		delete($prog->{qualifiers}->{LastShowing});
	    }

	    if ( defined($prog->{qualifiers}->{Subtitles}) ) {
		$writer->startTag('subtitles', 'type' => 'onscreen');
		$writer->dataElement('language', $prog->{qualifiers}->{Subtitles});
		$writer->endTag('subtitles');
		delete($prog->{qualifiers}->{Subtitles});
	    }
	    elsif ( defined($prog->{qualifiers}->{ClosedCaptioned}) ) {
		$writer->emptyTag('subtitles', 'type' => "teletext");
		delete($prog->{qualifiers}->{ClosedCaptioned});
	    }
	    if ( scalar(keys %{$prog->{qualifiers}}) == 0 ) {
		delete($prog->{qualifiers});
	    }
	}
	if ( defined($prog->{ratings_VCHIP}) ) {
	    $writer->startTag('rating', system => 'VCHIP');
	    if ( defined($prog->{ratings_VCHIP_Expanded}) ) {
		$writer->dataElement('value', "$prog->{ratings_VCHIP} $prog->{ratings_VCHIP_Expanded}");
		delete($prog->{ratings_VCHIP_Expanded});
	    }
	    else {
		$writer->dataElement('value', $prog->{ratings_VCHIP});
	    }
	    $writer->endTag('rating');
	    delete($prog->{ratings_VCHIP});
	}
	if ( defined($prog->{ratings_MPAA}) ) {
	    $writer->startTag('rating', 'system' =>'MPAA');
	    $writer->dataElement('value', $prog->{ratings_MPAA});
	    $writer->endTag('rating');
	    delete($prog->{ratings_MPAA});
	}
	if ( defined($prog->{ratings_ESRB}) ) {
	    $writer->startTag('rating', 'system' => 'ESRB');
	    $writer->dataElement('value', $prog->{ratings_ESRB});
	    $writer->endTag('rating');
	    delete($prog->{ratings_ESRB});
	}
	if ( defined($prog->{ratings_Warnings}) ) {
	    my %hash;
	    foreach my $k ( sort @{$prog->{ratings_Warnings}}) {
		$hash{$k}=1;
	    }
	    foreach my $k ( keys %hash ) {
		$writer->startTag('rating', 'system' => 'General Warning');
		$writer->dataElement('value', $k);
		$writer->endTag('rating');
	    }
	    delete($prog->{ratings_Warnings});
	}
	if ( defined($prog->{star_rating}) ) {
	    # comes in the form of a rating out fraction X/Y (X out of Y)
	    $writer->startTag('star-rating');
	    $writer->dataElement('value', $prog->{star_rating});
	    $writer->endTag('star-rating');
	    delete($prog->{star_rating});
	}

	if ( scalar(keys %$prog) != 0 ) {
	    if ( $opt_debuglistings ) {
		$writer->comment("\n   Left Over keys:".XMLTV::ZapListings::dumpMe($prog));
	    }
	    if ( $opt_debug ) {
		debugMessage("after write: prog:".XMLTV::ZapListings::dumpMe($prog)."\n");
	    }
	}
	$writer->endTag('programme');
    }
    return(scalar(@programs));
}

# write out a single channel in xml format
sub writeOutChannel($$$)
{
    my ($config, $writer, $station)=@_;

    if ( defined($config->getStationTransientFlag($station, 'notavailable')) ) {
	return(0);
    }
    
    if ( !$config->stationIncluded($station)) {
	return(0);
    }
    
    my $number=$config->getStationChannelNumber($station);
    my $letters=$config->getStationChannelCallLetters($station);

    if ( defined($number) && defined($letters) ) {
	$writer->startTag('channel', id=> "C".$number.lc($letters).".zap2it.com");
    }
    else {
	$writer->startTag('channel', id=> $station);
    }

    # just for testing - may be enabled (properly supported) in a future release 
    if ( 0 ) {
	if ( defined($config->getStationTransientFlag($station, 'zap2it-id')) ) {
	    $writer->emptyTag('channel-id', system=>"TMSID",
			      id=>$config->getStationTransientFlag($station, 'zap2it-id'));
	}
    }
    
    # just for testing - may be enabled (properly supported) in a future release 
    if ( 0 ) {
	if ( defined($number) ) {
	    $writer->dataElement('channel-number', $number);
	}
	if ( defined($letters) ) {
	    $writer->dataElement('channel-callletters', $letters);
	}
    }
    
    if ( defined($number) && defined($letters) ) {
	$writer->dataElement('display-name', "$number $letters");
	$writer->dataElement('display-name', "$number");
    }
    else {
	$writer->dataElement('display-name', "$station");
    }

    if ( defined($config->stationIcon($station)) ) {
	# Write as empty tag because icon can never contain anything.
	$writer->emptyTag('icon', 'src'=>$config->stationIcon($station));
    }
    
    # not supported as of yet
    #$writer->dataElement('url', $channel->{url}) if ( defined($channel->{url}) );
    $writer->endTag('channel');
    return(1);
}


# write the channels in xml format
sub writeOutChannels($$)
{
    my ($config, $writer)=@_;

    my $count=0;
    foreach my $station ($config->stationsInDisplayOrder()) {
	$count+=writeOutChannel($config, $writer, $station);
    }


    return($count);
}

1;
