The Company

Big Time Bank (BTB) is a large money center bank with branches and subsidiaries throughout the US as well as around the world. In the early days of bank deregulation, BTB had followed a philosophy of "let a thousand flowers bloom." Each group in the bank was allowed to focus on a target market and do what ever it felt was necessary to make money in its target market. Senior management managed these diverse groups through various financial controls, such as defined financial targets (i.e. assets under management, revenue) for each group as well capital allocations among the groups.

The resulting organizational structure was a bunch of independent, geographically focused (because of the banking laws), product oriented groups. There was little sharing of any type between these groups. Each group had a Business Manager who was responsible for all of the activities of the group. The Business Manager usually had a Chief Technologist who reported directly to him/her and who was responsible for managing a data center, the data center operations staff as well as a development organization for the benefit of the group. BTB acted very much like a conglomerate and the groups were treated like the subsidiaries held by a conglomerate.

The Private Banking Group

In the late 80s, the Private Banking Group (PBG) was formed from a few smaller units. Its mission was to focus on serving the needs of the "high net worth" clients of the bank.

The Western Hemisphere Division had been serving the needs of non-US clients in the Americas. Its programmers had developed a system that supported multi-currency Certificates of Deposit - the main product desired by these clients. This custom system ran on an HP minicomputer.

The US Division was created by selling various banking and investment products to the wealthiest clients from the existing Consumer Group client base. In addition to the basic consumer checking, saving and credit card accounts (provided through a special agreement with the existing Consumer Group) the US Division bought a securities processing package and offered various special mutual funds to its client base. The securities processing package ran on a mainframe in an MVS/CICS/VSAM environment.

The Legal Division had been serving the needs of law firms. The main product was an Escrow account management system that complied with the various legal requirements for such accounts. The main benefit for the clients was that the bank produced all the account statements that the law firm was legally required to provide to its escrow clients. The system was built on top of a banking package (HOGAN) and ran on a mainframe in an MVS/CICS/VSAM environment.

After a few years spent getting the PBG organized, the PBG management realized that the expected synergies and product cross-selling that were the rationale for the creation of the PBG were not occurring. Management felt that a significant part of the problem was that the systems used by the various divisions were on totally different platforms. In order for bankers to offer all of the products to all of the PBG clients bankers would need to be trained on multiple systems and multiple incompatible networks would need to be installed.

In order to correct this problem, management spent significant amounts of time and money to develop a system that would provide a unified view into all of the product systems (some of which were not even "owned" by the PBG). The final design was for all the product processing systems to feed an "integration system" (built on the HOGAN system) on a nightly basis a list of all the account balances and all of the transactions for the PBG accounts that were in the various product processing systems.

Once this system was in place, users were able to get a unified view of their clients’ relationship with the bank. However, they were unable to initiate transactions directly to the accounts that they were viewing. Instead, they filled out paper forms (different ones for each transaction), routed them past managers for authorizing signatures and sent them by messenger to a centralized back-office processing area. Specialists for each system would then enter the transactions into the appropriate product processing systems and in due course they would show up in the integrated account view.

When transactions crossed system boundaries, each system would post one side of the accounting for the transaction to a General Ledger System (GL) "wash account". For example, if a client wanted to buy a foreign currency CD with funds from his checking account then the withdrawal from the checking account would be made in the HOGAN system and the funds transferred to the GL account. Then the CD system would withdraw the money from the GL account and open the CD with those funds. When both systems finished their daily posting, the balance in the wash account should have been zero. Unfortunately, transactions were not always completed within one day on both systems. As a result, about 10 people (the proof and control clerks from each area) stayed until 7PM daily reconciling all of the differences.

An analysis of this process showed that transactions took hours to execute - mostly waiting for authorization sign-offs or for messengers to carry them from the various out-boxes to the back-office processing area. It was also determined that most of the daily proof and control problems were created by transactions that arrived too late to be entered into one of the systems involved in the transaction. In light of this analysis, management decided to build a system that would automate the transaction request process.

During the analysis, it was discovered that the different divisions had different authorization policies. These policies varied by, originating unit (i.e. branch), transaction amount and transaction type. The variation was primarily in how many people had to sign-off on a transaction and the organizational relationship of the authorizer to the person who typed in the transaction. The analysis also showed that all transactions went through the following phases:

• Transaction entry, in which the information about the transaction was collected.
• Transaction verification, in which a second person would verify that the client actually issued the instructions and that what was entered reflected the client’s wishes.
• Transaction authorization, in which various officer(s) signed-off on the transaction.
• Transaction execution, in which the client’s instructions were actually carried out.

Since most of the systems were not under PBG control (and those that were under PBG control were mostly on different platforms) it was decided to leave the back-office specialists, who would enter the transactions on the various product processing systems, in place. The system would just be used to replace the messengers and the routing of the paper transactions.

The Original System Design

The system was developed in the HOGAN environment on the mainframe because it had all of the information collected by the "integration system". This allowed the transaction request system to fill in various fields (such as account titles) on each transaction request and to validate other fields (such as balances). As a result, the system was able to reduce the data entry requirements for the users.

A language for describing authorization rules and an authorization engine to execute these rules were developed. Since each transaction only required limited information, it could fit on to one screen. Once the initial data entry was completed, the data was frozen and users could only approve or reject a transaction. Each approval caused a state transition in the authorization engine which was recorded in the transaction history. The approvals continued until the transaction was ready for execution or was rejected. When a transaction received its final authorization, it was printed at the station of the back-office specialist who was responsible for that product processing system. The specialist would then enter the information into the proper system and record the results on the printout (which would be kept for audit and proof purposes). Since both the transaction routing system and the HOGAN application system were running in the same environment, if a transaction was destined for a HOGAN account and received its final authorization the transaction would be executed immediately.

In addition, the system allowed all the users to see summaries of transactions by status, originating unit and current processing unit. This allowed managers and back-office specialists to monitor expected work and to call authorizing units that were slow in doing their authorizations (which was likely to cause problems with proof and control because of late transactions).

The results were quite satisfactory. Transactions that had previously taken several hours to execute, because papers were waiting for messengers to pick them up, were now being executed in under a half hour. Also, the proof and control clerks were finally able to get out most evenings by 5:30 PM instead of 7:00 PM.

Phase II - Extending the system

Although the system was working as designed, management was still not happy. As the PBG continued to expand, it was necessary to add back-office processing clerks to handle the growth in transaction volume. This growth in staff meant that the PBG fixed costs were too high - yet the clerks were needed in order to glue together all of the systems that did not communicate with one another.

A review of the system revealed the following problems:

1. While the system routed the transactions between the various front-office authorizers and the back-office processing clerks, once the transaction reached the back-office the transaction was printed out and the old paper-based process was used. This worked because all of the back-office clerks were located on the same floor in the processing center and routing paper was not too difficult in this environment.
2. Most of the errors that were being uncovered (once the system was installed) were typos in reentering transactions from paper into the product processing system.
3. The Legal Division clients (law firms) were initiating wire transfers from their Escrow accounts while they were sitting with their clients and completing the transactions (such as buying office buildings) that the money was set aside for. They needed to know when the funds transfer was completed via Fedwire and what the Fedwire reference number of the money transfer was in order for the counterparty (such as the seller of the building) to verify that it had actually received the money. The existing system would only show the bankers that the transaction had reached the back-office. It did not indicate when the wire transfer was complete or what the reference number was. This information could only be gotten by calling the back-office specialist and having them look at the transaction in the wire transfer product processing system.
4. Foreign exchange (FX) transactions HAD to be completed two business days (usually) after the transaction was executed because of market standards. (Normal settlement of an FX transaction occurs two business days after the trade. This allows enough time to verify the trade, transfer funds etc.) In terms of the product systems this meant that the funds needed to be withdrawn from the client account on trade date and deposited into the FX system clearing account (via the GL). Two days later, the currency bought would be deposited by the FX clearing system into a second clearing account from which the system would send the money to their ultimate destination as specified by the client. Yet the current system considered these transactions complete when they were printed in the back-office processing area (on trade date).
5. The authorization process was working so well that the bankers wanted to add various non-financial transactions to the transaction processing system. One particularly important non-financial transaction that needed authorization was address changes. Past experience had shown that embezzlers would try to change the statement mailing address so they could substitute a statement that did not show the embezzling transactions.

For these reasons, it was decided to enhance the system. The goals were to provide system to system linkages between the transaction processing system and the various product processing systems. The linkage to the wire transfer system was chosen as the first to be implemented because of the Legal Division needs.

The Modified System Design

Further analysis of the problem showed that the inter-system communications links were prone to failure. The design had to be able to accommodate these failures without causing the final authorization to fail if the only problem was a communications link failure. It was decided that each inter-system link would have an asynchronous process (a daemon) that would only take transactions destined for a particular product processing system (such as wire transfer) that had received their final approval and send them on to the processing system. The process would be started on a scheduled basis (a chrontab). New states were added for the various stages of this process, such as:

• Sent to product processing system (this would collect a reference number assigned by the product processing system and place it in the transaction)
• Rejected by product processing system
• Processing completed by product processing system (this would also collect some additional information - such as the Fedwire number)

Several new management/monitoring facilities were built so that the back-office operational area could control these new facilities. A link monitoring an management system was built to allow the back-office area to monitor the status of the link. The back-office users were also able to maintain the frequency with which the deamon would be started (which varied by time of day).

Problems Encountered in the Implementation

The biggest problem encountered during the implementation was that knowledge of the valid transaction states was shared among several components. The approval mechanism (which caused the state transitions), individual transaction types (which "knew" when to execute the transaction based on the fact that status was "final") and the workload monitoring system (which displayed transactions by state etc.) all needed to be modified to handle the new states. They also all needed special code to handle the fact that not all transactions had these new states.

The implementors also began to recognize that what were once considered separate transactions were really various sub-transactions that were combined in different ways to create "transactions". For example, a client could request:

• Withdraw money from my account and wire it to an account at another financial institution (i.e. my child’s account).
• Withdraw money from my account and mail a bank check to someone (i.e. pay my American Express bill)
• Withdraw money from my account, convert it to another currency (do an FX) and wire it to an account at another financial institution (i.e. allow my business to pay a foreign supplier).
• Withdraw money from my account, convert it to another currency (do an FX) and deposit the funds to another account at the bank (i.e. the client wants to shift funds to another currency).

The common components were recoded in each transaction. It would have been better to extract the sub-transactions (methods of getting funds out of various accounts, methods of sending funds to various internal and external systems) and reuse them in all of the transactions.

Copyright June, 1997 by Shalom Reich sreich@acm.org
Permission is granted to use this for personal use or any eduactional purpose as long as students are not charged for this material and this notice is reproduced together with the material.
All other users should get written permission from the author.