----------------------------------------------------------------------

Date: Fri 13 Sep 85 00:22:19-PDT
From: Peter G. Neumann <Neumann@SRI-CSLA.ARPA>
Subject: Some financial disaster cases from Software Engineering Notes
To: RISKS@SRI-CSLA.ARPA

I hope that the RISKS Forum will not degenerate into only an SDI Forum, so I
thought I would counterbalance this issue with a new topic.  I have
resurrected a contribution from the July 1985 SIGSOFT SEN, and also preview
some newer cases that will appear in the October 1985 SEN (which is just
about ready to go to press).  (The few of you who are ACM SIGSOFT members
please pardon me for the duplications.)

                         -------

     [FROM ACM Software Engineering Notes vol 10 no 3, July 1985]

Disasters Anonymous 1: A Rose is Arose is (Three) Z-Rose

Now and then I get a story that I cannot print.  (I do have a few, but don't
ask.  I have of course conveniently forgotten them all.)  Here, is one that can
be printed -- although its author must remain anonymous.  Note that the case of
the three extra zeroes resulting from two different assumptions about the human
interface bears an eerie resemblance in cause to the case of the shuttle laser
experiment, which follows after this one.  [PGN]

  A group within my company had a policy of dealing only in multiples
  of one thousand dollars, so they left off the last three digits in
  correspondence to the wire transfer area to make their job easier.
  Other groups, however, had to write out the full amount since they did
  not always deal with such nice round numbers.  One day, a transaction
  was processed that had a value of $500,000.  The person who entered the
  transaction thought that it was from the group who dealt in multiples
  of $1000 and entered it as $500,000,000.  Of course, this was not the case,
  so a $500,000 transaction became a $500,000,000 one.

  The only thing that prevented a disaster was that it was sent to a small
  company that called back to verify the amount, and the error was then
  caught.  However, this was a Federal Reserve transaction and the funds
  had been transferred, but the timing was good and the transaction was
  backed out before it became a disaster.  My opinion is that such critical
  software should have caught the error before the wire was sent to the
  Federal Reserve.

  Another error in a Federal Reserve transfer had to do with multiple
  transactions per communications transfer.  In this case, the Federal
  Reserve software put a pair of nulls in the data that should have been
  translated as blanks.  However, they were stripped out and a $200,000,000
  incoming wire lost.  To maintain the Fed balance, money was purchased
  to cover a deficit that didn't exist -- since the money was a credit.
  This was a substantial monetary loss because of inadequately tested
  software.

                         -------

     [FROM ACM Software Engineering Notes vol 10 no 5, October 1985]

Disasters Anonymous 2: Financial Losses

Our anonymous contributor from SEN 10 3 (July 1985) has come through again.

  Since I sent some disaster reports to you in May, another one has occurred.
  This one caused some financial loss and acute headaches among managers.

  Most large banks subscribe to the Federal Reserve's funds transfer system,
  frequently referred to as "Bankwire".  Our system that connects to Fedwire
  was being upgraded with a new DDA interface to the host to help protect
  against overdrafts.  During a review, it was determined that the software
  was not quite ready, but should be okay to put into production two days
  later.  I cautioned them against doing so since not all of the bugs had been
  resolved, and the software had not been "stress tested" (or whatever phrase
  you wish to use about testing that ensures that it will work in production).

  The first day of production went fine.  However, the main file in the new
  software was an ISAM file that had degraded significantly during the first
  day.  On the second day, that file continued to fragment and started to
  consume a large amount of the system resources.  This slowed response time
  so much that by the end of the banking day, we still had hundreds of wires
  to send to the Federal Reserve.  We had to request extensions every half
  hour for hours to try and squeeze the transactions through the system so
  that the money would get to our customers.

  In addition, the response-time problem and other bugs in the software
  prevented us from knowing our Federal Reserve balance.  Since we must
  maintain some 150 million dollars in our Fed "checking account", this lack
  of information could cause significant financial loss as 1.5 billion dolars
  were posted that day and we were off by hundreds of millions of dollars at
  first.

  Another part of this disaster is that the slow response time caused one
  program to assume that the host was down.  When a transaction finally went
  through, our system would transmit the DDA information, but the host did not
  acknowledge that they already had the wire.  Thus a large number of wires
  were being "double posted" (money sent twice).  At the end of the day, tens
  of millions had been double posted.

  As of this writing, the Fed balance had been straightened out, but not all
  of the double postings had been recovered.  Note that at current interest
  rates, a bank loses $350 per day per million dollars of unused money.

                         -------

     [FROM ACM Software Engineering Notes vol 10 no 5, October 1985]

Disasters Anonymous 3: Insurance, Reinsurance, and Rereinsurance

Perhaps anonymity is contagious.  Re: reinsurance, here is
another letter from a different contributor.

  I'm newly receiving SEN and found the ``war stories'' quite interesting.
  Here are three more.  I would prefer anonymity should you choose to print
  these.

  This first is hearsay (from a former co-worker).  Apparently he and his
  wife had a joint account with a $300 balance.  They needed $200 in cash, but
  due to miscommunication they both made $200 withdrawals - she at a teller's
  window (cage?) and he at an ATM (automatic teller machine) - within minutes
  of each other.  When the dust settled they found that their account had a
  zero balance:  the first $200 withdrawal left a $100 balance, the second
  should have left a negative balance of $100, but the computer generated a
  $100 credit to offset the shortfall.  The icing on the cake was my friend's
  inability to explain/convince the bank of this situation and have them accept
  restitution.

  I need to be circumspect about this second story -- it might well have
  involved fraud.  While a consultant, I was hired to review a reinsurance
  agreement.  The reinsurance industry is an old-boys, ``handshake is my bond''
  industry as insurors frequently offset their risk by selling it (reinsuring)
  to other insurors.  That is, I insure your building for $10,000,000 and
  re-sell all or part of that risk to another firm.  Apparently, late one
  Monday morning (nearly 11:00 a.m. EST), my client got notice across his
  computer network from another firm that it was reinsuring (i.e. off-loading
  risk) to my client to the tune of several million dollars.  The message was
  time-dated Friday evening (6:00 P.M., WST).  As ``luck'' would have it the
  property in question had suffered a catastrophic loss over the weekend.  The
  bottom line was that the message had been sent directly (not through any of
  the store-and-forward services) and the time-date was thus determined by the
  clock-calendar on the sender's computer.  Need I say more?

  Finally, a story told to me ``out of school'' by a friend at one of the
  nation's largest insurance companies.  They apparently are involved in so
  many reinsurance deals that it turned out that they were reinsuring
  themselves.  I.e., Jones reinsured with Smith who reinsured with Brown who
  reinsured with White who reinsured with Smith.  Smith, it turned out was
  paying both Brown and White commissions for accepting his own risk.  The
  computer system was not designed to look beyond the current customer,
  neglecting the loop.

----------------------------------------------------------------------
----------------------------------------------------------------------

Date: Fri 29 Nov 85 00:43:47-EST
From: Peter G. Trei <OC.TREI@CU20B.COLUMBIA.EDU>
Subject: Computer snafu halts treasury
To: risks@SRI-CSL.ARPA

       From the Wall Street Journal,  Monday 25 November 1985
                     [quoted without permission]

       A Computer Snafu Snarls the Handling of Treasury Issues
               by Phillip L. Zweig and Allanna Sullivan
              Staff reporters of the Wall Street Journal

   NEW YORK- A computer malfunction at Bank of New York brought the
Treasury bond market's deliveries and payments systems to a near-
standstill for almost 28 hours Thursday and Friday.
   Although bond prices weren't affected, metal traders bid up the
price of platinum futures Friday in the belief that a financial crisis
had struck the Treasury bond market. However, Bank of New York's
problems appeared to be more electronic than financial.
   The foul-up temporarily prevented the bank, the nation's largest
clearer of government securities, from delivering securities to buyers
and making payments to sellers - a service it performs for scores of
securities dealers and other banks.
   The malfunction was cleared up at 12:30 p.m. EST Friday, and an
hour later the bank resumed delivery of securities.
   But Thursday the bank, a unit of Bank of New York Co., had to
borrow a record $20 billion from the Federal Reserve Bank of New York
so it could pay for securities received. The borrowing is said to be
the largest discount window borrowing ever from the Federal Reserve
System. Bank of New York repaid the loan Friday, Martha Dinnerstein, a
senior vice president, said.
   Although Bank of New York incurred an estimated $4 million interest
expense on the borrowing, the bank said any impact on its net income
"will not be material." For the first nine months this year, earnings
totaled $96.7 million. Bank of New York stock closed Friday at
$45.125, off 25 cents from the Thursday, as 16,500 shares changed
hands in composite trading on the New York Stock Exchange.
   Bank of New York said that it had paid for the cost of carrying the
securities so its customers wouldn't lose any interest.
   Bank of New York's inability to accept payments temporarily left
other banks with $20 billion on their hands. This diminished the need
of many banks to borrow from others in the federal funds market. Banks
use the market for federal funds, which are reserves that banks lend
each other, for short-term funding of certain operations. The cash
glut caused the federal funds rate to plummet to 5.5% from 8.375%
early Thursday.
   The electronic snafu is by far the largest of computer problems
that periodically have bedeviled the capital markets.
   Almost all goverment securities transactions are settled
electronically through the New York Federal Reserve Bank. In this
system, computers of clearing banks are linked to one another through
a central computer, to enable banks to settle purchases and sales of
securities by customers.
   According to Wall Street sources, the malfunction occurred at 10
a.m.  Thursday as Bank of New York was preparing to change software in
a computer system and begin the days operations. Until Friday
afternoon, Bank of New York received billions of dollars in securities
that it couldn't deliver to buyers. The Fed settlement system, which
officially closes at 2:30 p.m., remained open until 1:30 a.m. Friday
in the expectation that technicians would be able to solve the
problem.
   Rumors about bank problems often send commodity traders scurrying
to buy precious metals. In the platinum pit at the New York Mercantile
Exchange, the price for January delivery surged $12.40 an ounce to
$351.20 Friday on volume of 11,929 contracts, a 29-year record.
   Reports that the Fed was investigating transfer problems at Bank of
New York prompted the platinum buying.

                          [end of quotation]

   I talked to a friend of mine who was peripherally involved in the
recovery from this 'snafu', and it seems that the primary error
occured in a messaging system which buffered messages going in and out
of the bank.  The actual error was an overflow in a counter which was
only 16 bits wide, instead of the usual 32. This caused a message
database to become corrupted.  The programmers and operators, working
under tremendous pressure to solve the problem quickly, accidently
copied the corrupt copy of the database over the backup, instead of
the other way around.

     One thing I have often noticed is that the 'normal run' code of
software packages tends to get much more through testing then the
code for error recovery; not only is it more difficult to test, but
the general feeling of 'this code will never execute' demotivates
programmers. In this case, it sounds like the people at BONY never
held a 'fire drill' to figure out how to handle a corrupt primary
database. Does anyone else have examples where attempts at error
recovery magnified problems?
       Peter Trei
       oc.trei@cu20b

----------------------------------------------------------------------
----------------------------------------------------------------------

Date: Wed, 18 Dec 85 15:47:45 est
From: friend@nrl-csr (Al Friend)
To: RISKS@SRI-CSL.ARPA
Subject: $32 Billion Overdraft

From: Al Friend - SPAWAR
To: Risks Forum

  Here is an interesting article out of the Washington Post.  It seems that
there is a very real potential for financial disaster lurking in the
electronic banking jungle:

[Washington Post, 13 December 1985, p. D7]

Computer Snarled N.Y. Bank
--------------------------
$32 Billion Overdraft Resulted From Snafu
-----------------------------------------
By John M. Berry, Washington Post Staff Writer
----------------------------------------------

  The Bank of New York, the nation's 18th largest, had a brief $32 billion
overdraft on its cash account at the New York Federal Reserve Bank when a
computer failure last month snarled thousands of government securities
transactions, a congressional committee was told yesterday.
  By the end of the day, the overdraft had been reduced to $24 billion, and
the bank actually had to borrow that amount from the New York Fed -- pledging
all of its assets -- in order to balance its accounts overnight.
  Aside from the unprecedented scale of the borrowing, and the spillover
effects on the government securities market, the incident intensified concern
at the Federal Reserve over the vulnerability of the nation's financial
payments system to a technological glitch that could have disastrous
consequences.
  Federal Reserve Chairman Paul A. Volcker and New York Fed President
E. Gerald Corrigan went before a House Banking subcommittee yesterday to
describe how the computer failure occurred and how the Fed and the bank
dealt with the crisis it caused.
  On Wednesday, Nov. 20, transactions involving more than 32,000 different
government securities issues poured into the Bank of New York, one of the
largest processors of such deals on behalf of others.
  The bank's computer system was supposed to be able to cope with up to
36,000 issues, but a programming glitch developed and, unknown to anyone,
the computer began to "corrupt" the transactions and make it impossible
for the bank to keep them straight.
  Because of the computer system breakdown, the bank could not instruct
New York Fed where to send the securities arriving at the Fed on behalf of
the bank's clients, and therefore could not get paid for them.  The New
York Fed was automatically taking money out of the Bank of New York's cash
account to pay the sellers for the incoming securities, all of which are
represented simply by computer records, rather than the familiar paper
bonds still used by most corporations.
  By Thursday evening, as hundreds of employes at a host of banks and
government securities dealers tried to sort out the problems caused by the
failure of the intricate and largely automatic network handling these
transactions, the bank had a $32 billion overdraft on its cash account at the
New York Federal Reserve Bank.
  The bank's computer specialists finally came up with a "patch" for its
 computer program -- a process described yesterday by its chairman,
J. Carter Bacot, as the electronic equivalent of patching a tire -- that
allowed it to begin to clear some of the backlog.  But just after midnight,
the patch failed too, after the overdraft had been whittled down to about
$24 billion.
  The Fed kept both its nationwide wires for securities and cash transactions
open in the early hours of Friday morning.  When the patch failed, the Bank
of New York was still able to borrow $700 million from other banks.  The rest
was covered by a $23.6 billion loan from the New York Fed.  As collateral,
the bank pledged all its domestic assets and all its customers' securities
it was allowed to use for such purposes.  Altogether, the collateral was
worth #36 billion, according to the Fed.
  The drama was not over.  Around 5 a.m. Friday, the bank finally completed
reconstruction of its customers' transactions from Wednesday.  By 10 a.m.,
it had done the same for the Thursday deals.  But, meanwhile, the rest of the
government securities industry had begun its Friday activities, and securities
and an overdraft were piling up again in the Bank of New York's account at the
New York Fed.
  "Faced with this situation," New York Fed President Corrigan told the
banking subcommittee, "at about 11:30 a.m., we temporarily stopped accepting
securities transfers for the account of Bank of New York in an attempt to
stabilize the situation somewhat and to see whether it was practical to
prevent further increases in the overdraft without causing excessive
disruption in the market more generally. . . .
  "Operationally, this meant that holders of government securities who had
contracts to deliver those securities . . . to the Bank of New York for
one of its customers [in return for payment] were temporarily unable to make
delivery under those contracts," Corrigan said.
  The stoppage lasted only for about 90 minutes that afternoon, and news of it
did not spread widely for nearly an hour.  Yet that disruption at the clearing
bank was enough, Corrigan said, to make some market participants unwilling to
trade securities among themselves.  "Perhaps most importantly, there was also
some evidence that investors were beginning to seek to break trades and
financing transactions with dealers serviced by the Bank of New York."
  Shortly after noon, the Bank of New York was able to begin handling the
Friday transactions that had been piling up, and the Fed was again able to
accept securities destined for the bank.  By that point the bank was operating
with a computer system that had undergone a major overhaul in less than 24
hours.
  The crisis was over, but its final bill is still mounting.
  The Bank of New York was out of pocket about $5 million, an amount equal to
about 7 percent of its earnings in the first nine months of this year, to pay
interest on the money it had to borrow that Thursday.
  It is still negotiating with many of the parties who may have sustained
losses in transactions that were not completed on time.  Such negotiations are
common, said an official of one major securities dealer, because a few
transactions are always going awry.  This time it was thousands.
  Some customers walked away in better shape.  "Indeed, those individuals and
intistutions who bought securities in question received a windfall in that
they received interest for a day [on the securities], but did not incur any
cost of financing," Corrigan noted.
  But any loss or gain in dollars, even with millions of dollars at stake, is
not the real issue.  What worries both Federal Reserve officials and
participants in the government securities market is the potential for a
failure of the system.
  On the average day, about $200 billion worth of government securities
transactions take place involving about 27,000 separate transactions, Corrigan
said.  Some days the totals are far larger.
  "Like it or not," Volcker told the subcommittee, "computers and their
software systems -- with the possibility of mechanical or human failure --
are an integral part of the payments mechanism.  The scale and speed of
transactions permit no other approach.
  "In the last analysis, no mechanical system can be entirely 'fail-safe'
and also be commercially viable," he said.  "The costs would simply be too
high, and the money and Treasury securities markets could not operate at the
present level of efficiency."
  The Fed chairman pointed out that, in this case, the Fed was available to
lend the $23.6 billion, on good collateral. "The effects in this instance
were of unprecedented magnitude, measured by the amount of the overnight
loan," he said.  "But the effects in terms of market performance and risk
were well contained. . . .  I believe it would be wrong to overdramatize this
incident."
  Corrigan in his more detailed testimony sounded more notes of concern.  "I
believe our actions were prudent, disciplined and appropriate.  In saying
this, I should also confess that in some respects we were a bit lucky," he
said.
  Part of the luck was that the bank was able to get its computer going again
as soon as it did.  Another part, Corrigan said, was that Thursday was not an
especially heavy day for securities transactions.
  One government securities trader summed up the situation this way.
  "We're all afraid something will go bump and send the market into a tailspin.
. . .  The Fed is working night and day to figure out what it can do.  The
banks are working night and day.  But the amount of [trading] in financial
markets is so large that we feel this is the No. 1 financial problem of the
next few months.  Banks have to be able to make settlements with each other."
 

----------------------------------------------------------------------
----------------------------------------------------------------------

Date: Sun 16 Feb 86 20:04:54-PST
From: Peter G. Neumann <Neumann@SRI-CSL.ARPA>
Subject: SF Federal Reserve Bank 2 Billion Dollar Goof
To: RISKS@SRI-CSL.ARPA

The SF Chronicle (7 Feb 86) had an article on what was "perhaps the biggest
banking blunder ever" (despite the Bank of New York just having had a $32
billion screw-up, reported in RISKS-1.31).  On 21 January 1986, the Fed was
testing its computers and accidentally transferred $2B to 19 financial
institutions.  A weekend test session had been constructed using 1000 actual
transactions from the previous Friday.  The test program and data were
accidentally left around, and thus the transactions were repeated on Monday
morning.  As opposed to the $32B case, all of the money was recovered, and
no actual losses were incurred.  A spokesman "stressed, however, that $2
billion represents only 2 percent of the funds handled by the Fed each day."
(... peanuts ... chicken-feed ...?)  In the future, testing will be done with
make-believe transactions and fictitious account numbers.  Six employees
deemed responsible were suspended without pay for three days.

   [Thanks to W. Randolph Franklin <wrf@degas.berkeley.edu> for reminding
    me of that one.  I had meant to include it earlier.  PGN]

----------------------------------------------------------------------
----------------------------------------------------------------------

From: ulysses!burl!rcj@ucbvax.berkeley.edu
Date: Sat, 8 Mar 86 20:45:11 est
To: ulysses!risks
Subject: bank robbery
Organization: AT&T Technologies @ Burlington, NC

I read an excellent book a few years ago simply entitled "Computer Crime".
                                       [PRESUMABLY BY DONN PARKER?  PGN]
I highly recommend it to the readers of mod.risks.  Here are a couple
of example horror stories from the book (from memory, sorry):

  a) A guy gets a bank loan, when he gets his payment book he sends in the
  *last* payment slip from the book with his first payment.  The bank's
  computer sends him a cheerful letter congratulating him on settling his
  debt in a timely manner.

  b) A guy opens an account at a major NYC bank with several thousand dollars.
  After he gets his personalized checks, he goes to a shady printer friend
  and has the guy print up identical checks but with a bogus magnetic number
  on the bottom.  He then goes on a $1,000,000 check-writing spree.  Every
  time on large purchases they call his bank and electronically verify that
  he can cover the check.  Every time the sorting machine at the bank sees
  the leading ?3?-digit code of a West Coast bank, and automatically mails
  the check there.  The West Coast bank's sorter kicks the check out to
  manual sorting because it has a bogus account number.  The human sorter
  takes one look at the check and sees the name of the NYC bank and blithely
  mails it back...  They finally got onto him when one of the checks had
  been through so many sorter and mailer machines it was nearly in shreds,
  and the human sorter on the West Coast got curious enough to look at the
  magnetic ink number.

  c) Guy opens an account in a Washington, D.C. bank.  He rips off several
  pads of blank deposit slips from the lobby of said bank, takes them to
  a location (?maybe he worked at the place?) that has a magnetic ink
  typewriter.  He laboriously types his own account number on the bottom
  of all the slips, then places the pads back in the lobby of the bank.
  A month later he withdraws $100,000 and disappears.

The MAD Programmer -- 919-228-3313 (Cornet 291)
alias: Curtis Jackson ...![ ihnp4 ulysses cbosgd mgnetp ]!burl!rcj
   ...![ ihnp4 cbosgd akgua masscomp ]!clyde!rcj
       [OLD STUFF, BUT WHY NOT?  WE HAVEN'T HAD THEM HERE BEFORE.  PGN]

----------------------------------------------------------------------