Enterprise Level Inbound and Outbound Faxing
The Problem
Consider the numbers involved. A customer needed to send out approximately 20,000 fax messages per week from a mainframe and middleware environment in a data centre but that data centre did not have voice channels, and the administrative effort to cajole the data centre managers into bringing ISDN 30 feeds right to a server or patching ISDN30 (Primary Rate Interface or PRI) feeds was too much of a burden for them to manage. Further more the customer receives approximately 15,000 faxes per week on a variety of devices spread over a headquarters building and two call centres. Common sense would have been to bring ISDN 30 feeds into the data centre and have the network provider reroute geographical and non geographical numbers for inbound faxes into the data centre where a single or pair of fax servers can process all the messages.
Unfortunately, as with many Outsourcing arrangements the common sense aspect does not apply, and the price for anything non standard or remotely innovative can go through the roof. A different solution had to be found that could utilise the huge number of ISDN trunks in the two call centres and giving the customer the option to slowly change their geographic phone numbers (i.e. 0207) into non geographic ones (0870 etc.) Furthermore, the data centre disaster recovery arrangements were themselves outsourced to another 3rd party so any solution located in the data centre needed to be replicated elsewhere.
Business Volumes
The business requirements gathering found up the following information:
Bulk messages weekly:
Outbound: 20,000 faxes ranging from 4 pages to 20 pages per message
Inbound: 15,000 faxes to a battery of fax machines, average 6 pages each
General messages weekly:
Inbound: approximately 5,000 pages per week
Outbound: approximately 3,000 pages per week
Additional Information:
• The business management and call centre team leaders would benefit from personal fax numbers to receive and process urgent information
• The legacy fax solution delayed outbound faxes by up to 20 hours
• The legacy fax solution printed inbound faxes that were then scanned into an image and workflow solution
• Many inbound fax messages were being lost
• Due to the delay in the automatic outbound service a great number of fax messages were being printed onto paper and manually faxed, resulting in the recipient receiving duplicate messages onto often rickety hardware
• A huge number of faxes actually failed because the database information was out of date and there was no administrative process to resolve these issues
The Solution: Product and Vendor
Leaving aside product evaluation and selection steps, the Message Manager application from Systems Solutions was chosen as the platform that could take the best Dialogic card available and handle the volume. Other solutions, including Rightfax and Faxination were excluded for a variety of reasons, mainly on what PRI hardware they could support at the time of selection. The vendor chosen to install, commission and support the application was also a key factor in choosing the solution. There is no point in buying a solution if the vendor that you have a strategic and good relationship with doesn’t want to support it and you have little confidence in the vendors who want to sell you another solution, no matter how good that other solution might be.
The Outbound Design
To recap, there was one data centre generating the bulk of the messages and two call centres where the bulk of the ISDN30 trunks were. The solution adopted was to leverage the Microsoft Exchange environment in the data centre to receive the bulk outbound solution. The middleware was instructed to send all messages to the Exchange servers, a standard pair of Network Load Balanced Front End systems. The middleware used the NLB host name, ensuring that middleware to initial delivery point resilience was available. The middleware could not queue messages for very long so it was important that all traffic was despatched quickly. An event sink on the Exchange server archived these SMTP messages for a period of two days in case of a problem with the fax servers. From the Exchange servers the messages were relayed to another host name, this time a pair of A records (identical name, different IP addresses) in DNS with low TTL. This ensured that the messages were balanced between the two sending fax servers.
The two fax servers were located in each of the call centres (250 miles apart) were linked to the data centre by a resilient (different BT exchanges using diverse ducts taking different routes into the server rooms) 8Mb ATM circuit which was underused for most of the day. The Image and Workflow scanning activities called for a large link to get images and data to the data centre by 9am but after this time the traffic much lower. Again, common sense would be to have used twin 4Mb links to reduce costs but the contract with the network provider (outsourced through yet another 3rd party) did not provide for such a solution. The two data centres were themselves linked by a dedicated 2Mb circuit, primarily for intra site voice overflow but the link was available for specific host to host IP traffic. At a hardware level the two fax servers were HP ProLiant ML370 systems, chosen over the ProLiant DL380 systems because the ML370 had better and more resilient airflow capabilities (they had six fans!), a key factor when the fax cards get very hot. The DL380 was simply felt to be too much of a risk to justify the £1000 per unit cost saving, given that the fax cards cost over £5,000 to replace. The initial licence purchased eight channels out of a possible 30 on the PRI card in each location. Faxes were processed and sent to the first available channel. Three channels were reserved for inbound traffic, leaving five channels to send outbound messages unless a 4th inbound message had taken one of the five shared channels.
The Message Manager application is able to monitor other Message Manager installations and load balance between the two. There are always times in a day where one server will be busier than the other, through faxes being queued due to engaged numbers, larger volumes of inbound faxes taking more channels than desirable or any other reason. If one server does start to fill with a back log of traffic the Message Manager application will send some of the overflow to the other server or servers within the organisation to even out the backlog. The ISDN trunk was fed into the Avaya Definity G3r switch within the server room to be routed out of the building. In the first instance the Server to Switch and the Switch to Public Network circuits were bonded on a one to one basis which ensured that the fax solution did not end up taking over voice channels. There were some serious configuration issues on the Switch which were in the process of being resolved and rationalised and the fax solution was considered an administrative risk that did not need to be taken when technology could be used to keep traffic discreet.
Faxes sent to engaged numbers were paused and a new message attempted. This enabled the fax servers to rotate through messages much more efficiently than the previous system which had no other option than to retry the current fax for the number of configured retries before abandoning. Successfully sent messages were despatched to an SMTP address which applied to a Public Folder. There was a local Exchange server in the same site as each Fax server which received the status messages and an attachment with the TIFF images of what was actually sent. This folder was replicated during out of office hours to the other servers in the environment. The TIFF image was included to enable call centre agents to handle any queries that may have arisen between that the data centre generated and what was actually sent down the wires. These messages were retained for one week through standard Public Folder age limits.
Unsuccessfully sent faxes were sent to a different SMTP address (Public Folder), again with the TIFF attached and a process was put in place to ensure that the reason for failure was traced and resolved. The fax solution was able to append status text so that the agents assigned to problem resolution could tell that the number was just engaged for longer than the number of retries (a very busy destination device) or if the number was faulty or the line discontinued. Those unsuccessful messages were resolved and the correct information entered into the mainframe to reduce the incidence of problems. One of the things noticed was that a large number of messages to the same destinations were failing for the same reasons; either the fax was too busy or that the number was discontinued and a lot of messages were trying to be sent to it. The agents contacted the recipients and either the number was corrected or a less busy fax number was given and in one case a new number and fax machine was introduced purely for the purpose of receiving faxes from the Message Manager system. Over time the number of failed faxes fell to a small number.
All that has been described so far is the outbound solution from data centre to fax servers and out to the public network. There was also the inevitable and occasional number of faxes that were generated by call centre and other agents, as well as administrative, non operational staff. Hitherto, these messages were sent over dedicated fax machines using analogue lines with DDI numbers attached to them, so that inbound faxes could be received on those devices if necessary. As part of the centre refresh operation some of these devices were replaced by Xerox multi function devices with the capability to scan, print and fax. When the main fax solution was being implemented the business took the opportunity to swap the fax modules for email modules. When a user had a sheet of paper that they needed to fax then were trained to scan the item in and email it to phone-number@fax.com. The “fax.com” address was chosen as quite the simplest solution to use and applied onto to messages sent from the Xerox printers. The Xerox devices were given the multi addressed A record as the SMTP server and messages were sent amongst all the other bulk messages. Confidential outbound messages that required a signature and filing were able to have a sending SMTP address applied to them. This enabled the successful messages to be advised to the inbox of the sender rather than the standard Public Folder structure.
The Inbound Design
Hitherto, inbound fax messages had primarily been sent to a range of fax machines, all of them slow which printed out the entire message before signalling their availability to receive another message. In addition, there were a large number of fax machines dotted around the call centre and headquarters, sending and receiving in an uncontrolled manner. The Message Manager solution was to assign all users who wanted one, a personal fax number in a non geographic range. Message Manager would receive the fax message with the destination number translated into five significant digits and would do an LDAP lookup on Active Directory “pager” field to find the SMTP address matching that account. The decision was taken not to search on the “facsimileTelephoneNumber” attribute because that field contained the full fax number whereas the “pager” attribute was given the five characters that the Switch presented as significant digits.
The bulk inbound solution was radically different. Rather than receive all the messages and either print them on one of the Xerox Multifunction Devices or email them to a mailbox/Public Folder the decision was to accelerate a process that was already planned but a long way off in the development programme. Message Manager can also send received faxes, as TIFF images to a network share. The bulk inbound faxes were all sent to a small range of known DDI numbers. These numbers were diverted away from the common trunks and onto those dedicated for the fax solution. This diversion was merely a short term measure while the literature was modified to show new non geographic numbers rather than the location specific numbers. From the network share the Image and Workflow solution was able to take the TIFF images and process them. On some documents the workflow was automated by the software reading a preconfigured form and taking the numbers it found in a given area of the screen. In other cases the messages were sent into a generic queue where a customer services support agent processed the form and placed it into the right workflow queue. Inbound faxes that were printed and then scanned in became purely digital, in the form of TIFF images. This enabled a number of scanners, costing £30,000 each to be decommissioned. These scanners were reaching the end of their useful life and without a realignment of the development programme they would have had to have all been replaced and their numbers augmented due to an increase in business coming through the centres. General business related inbound messages were sent to a group mailbox in the same manner as personal messages. Standard mailbox permissions were granted to those users who needed access to this group inbox.
Benefits
A number of benefits came out of this solution. First was the increase in outbound capability from seven legacy 9600 baud fax devices to 10 channels, rising to 16 after a purchase of four extra channel licences per server (increasing the dedicated inbound from three to four channels per server) These faxes were sent at 28.8 rather than 9.6, although the majority of faxes were sent at 14.4 but still an improvement over 9.6 baud. Because the waiting time for outbound faxes reduced from tens of hours to single digit minutes the volume of urgent manually printed and faxed messages were almost eliminated. Another benefit of the more reliable solution was that more people actually requested the information by fax which reduced the pressure on the postal fulfilment service to print and send urgent letters. It is important to note at this point that email would have been by far and above the most desirable solution for this, but the regulatory issues around this kind of traffic (bank related information) were unresolved.
Part of the configuration issues with the Definity Switch included the fact that the legacy fax machines actually used channels which encroached on the call centre inbound lines. Busy traffic reduced the number of channels available for callers quite significantly. Bringing in a solution whereby even ad-hoc messages were “emailed” to the fax servers and transmitted over the discreet trunk handed channels back to the call centre, reducing the number of times a day where number unobtainable messages occurred within the network. Over twenty fax machines were quite literally thrown into skips, saving a lot of money on paper, toner and maintenance contracts.
Lessons Learnt
First and foremost was the lesson that the programme took away was that infrastructure solutions could help solve a tactical problem by realigning the strategic plan. This solution wasted absolutely no money in short term fixes. The next lesson was that the volumes predicted by the business were wildly out and not merely because the business were unreliable, although that had always been the case in anything the business had predicted or supplied to date. The design already put an extra 25% fat into the capacity because of previous projects teaching the teams that the business numbers were unreliable. What had not been predicted was that as confidence grew in the system internally the agents would be able to offer faxes to customers and that those customers would accept the offer because they knew that the fax would be delivered in minutes rather than (literally) days. Going from 8 to 12 channels per server was done earlier than anticipated but this was better than going straight to 12 channels and finding that the solution was underused. Network traffic was a bit of an issue. At first the processed messages were sent to the Public Folders and replicated immediately. This caused an impact because outbound messages were being sent up the WAN at the same time as archived of all messages, inbound AND outbound were replicating around the network. Devoting five minutes more thought to that simple matter rather than telling the business how wonderful it would be that they could see what had been sent and what hadn’t is an obvious one to learn.
Return on Investment
On the cost side there was the cost and support of the servers, software and the PRI cards. On the saving side there was the cost and support of the fax devices, the scanners and the consumables. In addition there was the ability of the existing number of support agents to be reduced because they firstly needed to scan less to could process more TIFF images. Secondly the software being able to process more meant that the agents had less TIFF files that needed manual intervention. Thirdly the reduction in calls for manually and expedited outbound traffic enabled a reduction in the number of staff employed within the operational support area. These efficiencies enabled more work to be processed on less equipment by less people.
A rough assessment showed that the cost of the solution would have paid for all the infrastructure changes within 3 months and paid for the development costs within a further three months. These development costs were all costs that would have been incurred further down the programme in any case so the fax solution actually saved nine man months of development cost, simply by doing that work earlier. No discernable cost was incurred by bringing forward the development work due later in the programme at the expense of other development activities.