How to use ftp in a shell script

Bruce Ediger
Bruce Ediger's home page

Party on!
Continue using Mozilla's browser.

You might also consider using Linux. It's faster and has less malware.
Sometimes I want to FTP a file from one machine to another. Usually, I can do the
transfer interactively, but every so often, I would like to have a shell script do the file
transfer. This task has eluded me in the past, but I finally figured it out. I've not seen
this particular trick documented in the past, so I submit it for your approval.
The Problem
The problem I always encountered in scripting ftp transfers involved getting a
password to the ftp server. Typical ftp client programs under Unix, Linux, Solaris and
NetBSD all read the ftp password from/dev/tty.
Example (non-working) script
#!/bin/sh
HOST='ftp.users.qwest.net'
USER='yourid'
PASSWD='yourpw'
FILE='file.txt'

ftp $HOST <<END_SCRIPT
user $USER
$PASSWD
put $FILE
quit
END_SCRIPT
exit 0

The above script will just hang if run in the foreground (in an xterm), or if run in the
background (from a cron job), it will fail to perform the work of transferring file.txt.
/dev/tty names a strange, magic device. Each process (more strictly each process
group) has a different /dev/tty, and you can not naively make ftp clients read the
password from some non-magic, yet convenient source, like a "here document". When
run in an xterm, the script above appears to hang because it reads the password
from /dev/tty. The xterm constitutes the script's /dev/tty, so the script waits for
keyboard input.
Example Working Script
#!/bin/sh
HOST='ftp.users.qwest.net'
USER='yourid'
PASSWD='yourpw'
FILE='file.txt'

ftp -n $HOST <<END_SCRIPT
quote USER $USER
quote PASS $PASSWD
put $FILE
quit
END_SCRIPT
exit 0

The Tricks
Getting the password to the ftp server without having the ftp client program read the
password from /dev/tty requires two tricks:
1. Using the -n option on the ftp client program to prevent the ftp client from
trying to log in immediately. That way, the ftp client does not ask for a user ID
and password. No use of /dev/tty.
2. Use the ftp client program command quote to send user ID and password to the
ftp server.
Further Refinements
The above sh script will spew lots of ftp client output to standard output. Even if
everything works perfectly, the user running the above script will see lots of
incomprehensible text scrolling by quite rapidly. One refinement would send output to
different places:
ftp -n $HOST > /tmp/ftp.worked 2> /tmp/ftp.failed <<END_SCRIPT

One could further refine error handling by acting on the ftp client program's exit
status:
ftp -n $HOST > /tmp/ftp.worked 2> /tmp/ftp.failed <<END_SCRIPT
blah blah
END_SCRIPT

EXITSTATUS=$?

if [ $EXITSTATUS != "0" ]
then
# handle the error...
fi

Except that the above doesn't always work - most FTP clients always exit with a status
of 0. This leads to ugly "false negatives": the file transfer fails, but the script doesn't
detect the problem.
One way to verify that a file transfer took place - transfer it back:

#!/bin/sh

ftp -n << END_SCRIPT
open $1
user $2 $3
put $4
get $4 retrieval.$$
bye
END_SCRIPT

if [ -f retrieval.$$ ]
then
echo "FTP of $4 to $1 worked"
rm -f retrieval.$$
else
echo "FTP of $4 did not work"
fi
Regular FTPs there and back of large files can consume a lot of time.
Control of ftp by a shell script
One obvious improvement would have the ftp client program controlled by the shell
script. I don't think that would comprise an impossible task, but I also don't think that
it would have much value. Scripting ftp transfer using expect might cause you less
pain.

Alternative #1
I saw a second way of doing this in a usenet article:
#!/bin/sh
USER=userid
PASSWD=userpw
ftp -n f2dev <<SCRIPT
user $USER $PASSWD
binary
get some.file
quit
SCRIPT

It still uses the "-n" trick, but it sends user ID and password in the same "user"
command.

Alternative #2
Use a .netrc file
Linux, Unix and BSD users have the alternative of using a .netrc file. The ftp man
page documents the format of .netrc. To accomplish the task of using ftp in a shell
script you would have to fill out a.netrc file something like this:

machine something.else.com
login myid
password mypassword

ftp demands that .netrc not have group or world read or write permissions:

$ ls -l .netrc
-rw------- 1 bediger users 51 Dec 16 13:30 .netrc

Using a .netrc file has a few problems that may or may not prevent you from using it.
A shell scripkt that does FTP using .netrc is no longer self-contained. You have
to keep track of two files, which means that bugs can be less than obvious.
ftp reads it's user ID's .netrc. If you develop your script under a given user ID,
then put it in production under a second user ID, you have to coordinate .netrc
file contents between those two user IDs.

Alternative #3
Apparently, the Ckermit program from Columbia University understands FTP. You
could use Ckermit to script FTP transfers. This looks to have advantages and
disadvantages. On the "pro" side, it appears that Ckermit can exit on various errors,
like unknown user IDs, or bad passwords. On the "con" side, you have to have
Ckermit. I don't recall that it had a too onerous install, but it doesn't come with many
Linux distros these days, and it probably doesn't come with any vendor Unix.

$Id$