This is one of several pages relating to the history of the automatic totalizator, its invention in 1913, the inventor George Julius and the Australian company he founded in 1917 which became a monopoly (later an oligopoly) in this field. This page documents the early mechanical and electro mechanical totalizator installations and the Brisbane computer totalizator installation. This is a history only non commercial page. If you wish to start from the beginning then go to the index .
|Timelines - The Premier (Julius) totalisator|
Note - The Premier totalisator was the name given to Sir George Julius' invention.
|No of||Year of||Where Installed||No of|
|1||1913||Auckland Racing Club, Auckland N.Z.||30|
|2||1916||West Australian Trotting Association, Perth W.A.||8|
|3||1917||Queensland Turf Club, Brisbane Queensland (* see below )||24|
|4||1917||Australian Jockey Club, Sydney, N.S.W||150|
|5||1918||Canterbury Park Racing Club, Sydney N.S.W||40|
|6||1918||Rosehill Race Club, Sydney N.S.W.||43|
|7||1918||Newcastle Jockey Club, Newcastle N.S.W||30|
|8||1918||Auckland Racing Club, Auckland N.Z.||58|
|9||1918||Wallsend Jockey Club, Newcastle N.S.W||20|
|10||1920||Wellington Racing Club, Wellington N.Z.||32|
|11||1920||Wanganui Jockey Club, Wanganui N.Z.||27|
|12||1920||Feilding Jockey Club, Feilding N.Z.||32|
|13||1921||South Australian Jockey Club, Adelaide S.A.||34|
|14||1921||Canterbury Jockey Club, Christchurch N.Z.||40|
|15||1921||Manawatu Racing Club, Palmerston North N.Z.||32|
|16||1922||Auckland Racing Club, Auckland N.Z..||26|
|17||1922||Ceylon Turf Club, Colombo Ceylon||12|
|18||1922||West Australian Turf Club, Perth W.A||34|
|19||1923||Madras Race Club, Madras India||24|
|20||1923||Victoria Park Race Club, Sydney N.S.W.||25|
|21||1925||Australian Jockey Club, Warwick Farm N.S.W.,||45|
|22||1925||Western India Turf Club, Bombay India||126|
|23||1926||South Australian Jockey Club, Adelaide S.A.||30|
|24||1926||Singapore Turf Club, Singapore||16|
|25||1926||Rangoon Turf Club, Rangoon Burma||40|
|26||1926||Ceylon Turf Club, Colombo Ceylon||49|
|27||1926||Western India Turf Club, Bombay India||50|
|28||1927||Willows and Colwood Park Racecourses, Victoria B.C. Canada||10|
|29||1927||Perak Turf Club, Ipoh Federated Malay States||18|
|30||1928||Societe d'Encouragement pour l'amelioration des Races de Chevaux en France, Longchamps, Paris||273|
|31||1928||Willows and Colwood Park Racecourses, Victoria, B.C. Canada||10|
|32||1928||Selangor Turf Club, Kuala Lumpur Federated Malay States||23|
|33||1929||Galle Gymkhana Club, Galle Ceylon||30|
|34||1929||West Australian Trotting Association, Perth W.A.||38|
|35||1930||Greyhound Racing Association, Harringay London England||84|
|36||1930||Greyhound Racing Association, Edinburgh Scotland||27|
|37||1931||Victoria Racing Club, Flemington, Victoria||123|
|38||1931||Moonee Valley Racing Club, Melbourne Victoria||63|
|39||1931||Williamstown Racing Club, Williamstown, Victoria||43|
|40||1931||Victoria Amateur Turf Club, Caulfield, Victoria||105|
|41||1931||Greyhound Racing Association, White City London England||96|
|42||1931||Greyhound Racing Association, Newcastle-on-Tyne England||24|
|43||1931||Greyhound Racing Association, Cardiff Wales||48|
|44||1931||Reading Stadium, Reading England||17|
|45||1932||Leeds Greyhound Racing Association, Manchester England||15|
|46||1932||Greyhound Racing Association, Manchester England||31|
|47||1932||The Wembley Stadium, Wembley, London England||70|
|48||1932||Bristol Greyhound Racing Association, Bristol England||48|
|49||1932||Miami Jockey Club, Hialeah, Florida, U.S.A.||110|
|50||1933||Malta Greyhound Racing Company, Malta||48|
|51||1933||Victorian Trotting and Racing Association, Ascot, Melbourne, Victoria||28|
|52||1935||Greyhound Racing Association, White City London England upgrade||96|
|53||1935||The Wembley Stadium, Wembley, London England Upgrade||69|
|54||1935||Greyhound Racing Association, Harringay London England upgrade||68|
|55||1935||Greyhound Racing Association, Hall Green, Birmingham England||28|
|56||1935||Midland Greyhound Racing Co., Wolverhampton England||28|
|57||1935||White City Manchester England||35|
|58||1935||Reading Stadium, Reading England Upgrade||8|
|59||1935||Greyhound Racing Association, Manchester England upgrade||30|
|60||1935||Canterbury Jockey Club, Christchurch New Zealand||40|
|61||1935||Australian Jockey Club, Sydney, New South Wales||113|
|62||1936||Madras Race Club, Madras India||74|
|63||1936||Greyhound Racing Association, Powderhall, Edinburgh, Scotland||20|
|64||1936||Welsh White City, Cardiff, Wales||12|
|65||1936||Brough Park, Newcastle, England||8|
|66||1936||Bristol Greyhound Racing Association, Eastville England||24|
|67||1936||Reading Stadium, Reading England Upgrade||17|
|68||1936||Greyhound Racing Association, Leeds England||10|
|69||1936||Wellington Racing Club, Wellington N.Z. upgrade||50|
|70||1936||Mentone Turf Club, Melbourne, Victoria||30|
|71||1936||Epsom Turf Club, Melbourne, Victoria||30|
|72||1937||Philippine Racing Club, Manila, Philippine Islands||28|
|73||1937||Clairwood Turf Club, Durban, South Africa||38|
|74||1937||Manila Jockey Club, Philippine Islands||9|
Contemporary Note - A data network with 273 terminals in 1928 ?
Contemporary Note 2 - The installations above are documented up to 1937, probably because this document was created around that time. Julius totes continued to be manufactured and installed for a long time after this. If anyone has more information please email me at the address at the bottom of this page. Rio de Janeiro and Caracas are examples of latter-day Julius tote installations of the 1950s.
The Secretary of the Wellington Racing Club writes: "The Automatic Totalisator installed at Trentham has been severely tested during the last two years. The turnover has been exceptionally heavy, on one occasion 102,000 pounds being invested on eight races. It is safe to say that the public are well served, better than ever before, and they in common with the executive of the Club are satisfied that the machine is the best on the market at the present time."
The manager of the Rosehill Racecourse Co. Ltd., writes: "Two 'Premier' Totalisators are installed on the Rosehill Racecourse Company's racecourse at Rosehill, and were used for the first time on the 9th March, 1918. Since that date both machines have been used at all meetings held in connection with this Company, and have given entire satisfaction to both the betting public and the officials of the Racecourse Company. The machines have done everything claimed for them by the makers, and we have no hesitation in recommending them to any Racing Club."
Arthur V Kewney, the Secretary of the S.A. Jockey Club, Morphettville, Adelaide, S.A., writes: "I am directed to inform you that after a 12 months' trial the Committee are very satisfied with the working and the smooth running of the Premier Totalisator which your Company installed for this Club at Morphettville. The machine has fully demonstrated its capability to do all the work that was claimed for it. On Adelaide Cup Day the second time it was in use, the machine handled 36,668 pounds, and on the Cup itself 7,213 pounds 5 shillings was invested, both records for any one machine operating in this state.
Since the installation of the Premier Totalisator in May 1921, 13 race meetings have been held at Morphettville and 253,123 pounds 10 shillings has passed through the machine. "
The W.A. Trotting Association Perth reports: "The Totalisator Machine supplied by your Company has been running continuously on our Course since 29th January 1916. Your machine has proved popular with the public, and has given satisfactory results to my Association. The machine is the first of its type installed on any racecourse in Australia."
C.W.Cropper Secretary Australian Jockey Club wrote on 22 July 1922 "I have pleasure in stating that the "Premier" Automatic Totalisator Machines, which were installed at Randwick Racecourse and are operated by your Company on contract under the supervision of the Club, and which were first used at the Club's Spring Meeting, 1917, have, since their installation, given complete satisfaction. Their accuracy is unquestionable and the rapidity with which tickets are issued by the experienced operators leaves nothing to be desired."
The Secretary of the Wallsend Jockey Club advises: "This machine has been at work on our Course since Christmas, 1917, and has given the utmost satisfaction to my Committee and the general public. There has not been mistake or complaint as to correct workings since its installation, and we will be pleased to recommend your machine to enquirers."
The Secretary of the Newcastle Jockey Club writes: "I am directed to inform you that your Totalisator has given every satisfaction to my committee and its patrons."
The General Manager of the Canterbury Park Race Club writes: "The Automatic Totalisators Limited machines have been in operation on our Course, both in the Saddling Paddock and the Leger Reserve, at all our Race Meetings held since the 23rd February, 1918, and during the whole of that time they have given every satisfaction, no hitch or mis-working having taken place at any time."
|The Brisbane Project|
The Brisbane totalisator system first opened at the Gabba Greyhound track on Thursday 25 January 1979 and during the following four weeks five other installations were commissioned and 10 race meetings were conducted. Figures available for the first 28 meetings, showed an overall increase of 85% in turnover compared with the same meetings the previous year.
The Brisbane project represented two technical milestones for the company's computer totalisator design.
The Brisbane totalisator system was the first on course computer totalisator system for Brisbane and
used to calculate dividends for the whole state of Queensland.
ATL mobile computer tote trailer
The above image shows one of the computer trailers parked in its bay at Eagle Farm racecourse. Parked in front of it, to the left under the tree is one of the mobile Rolls Royce backup generators. The ATL logo with the red bar under the letters ATL is visible on the side of the generator trailer. The object on top of the trailer is the generator's muffler. These generators outlived the PDP11 systems and worked for decades thriving on neglect. The only things that went wrong were perishable items like radiator hoses. At the far left of the picture near the base of the tree, the extreme front left edge of the Bedford prime mover is visible.
The ATL Sell/Pay Totalisator System for the Brisbane Metropolitan and Ipswich Racing Industry consisted of two Mobile Computer Vans ( semi trailers ), each van consisting of a twin PDP11 /34 based totalisator system. In each van the twin PDP11 /34 computers in the totalisator configuration operated in "Hot Standby" mode, the term referring to the twin operation of system computers that are effectively load sharing the transaction inputs with any one computer having the capacity to control the entire system. This concept is one that does not incur significant cost penalties and ensures the following: The loss of one computer has no effect or time loss on the totalisator operation.PDP11 systems inside one of the computer vans
All bet transactions were recorded in duplicate files held on separate disc sub systems. Each set of files was updated by a separate computer to reduce to a minimum the chance of data corruption of the same record in both sets.
Each bet transaction on file was identified by a Ticket Serial Number which contained a key to the location in the bet file as to where the bet record was kept.
Communications between the computer system and the J22 terminals was via serial asynchronous polled, multidropped lines.
The DMX11 was a microprocessor controlled multiplexer, which relieved the transaction processors from the overhead associated with communication protocol by polling each terminal on line in sequence, handling data messages between terminals and computers, performing error checks on every message received and transmitted.
The Queensland Racing Circuit had 165 J22 Terminals which were moved between five different venues, servicing Gallops, Trotting and Greyhound racing. The terminals were high performance dual communication ported, micro processor controlled and designed to operate within a computer based totalisator system. The micro processor was a Motorola 6800. Transaction details appeared on the operators alphanumeric display as they were entered. A versatile dotmatrix printer printed the details of the transactions including a computer generated unique serial number onto the tickets in alphanumeric and code form. The ticket was issued through a chute in front of the customer's display.A ticket printed on a J22 Terminal
Winning tickets were fed into the terminal via a chute situated adjacent to the operators display. An opto-electronic reader scanned the bar code and transmitted the bet detail to the computer system for verification, the amount payable was transmitted back to the terminal and displayed to both the customer and the operator, the winning ticket serial number was cancelled, and all this was performed in under two seconds.
J22s in operation Guineas Room Eagle Farm 1979
|Memories of systems long gone|
The PDP11 tote Console Terminals
The console terminals above are the ones Colin would pounce on when there was a problem. The two hard-copy terminals are Teletype KSR43s. The two VDUs are Datatronics terminals, which are described later. Behind the photographer is a good sized workbench and above the workbench on the wall is the phone system which conveyed the tales of woe from the tote houses. The clanking disk drives mentioned above needs some explanation. These drives do not match the image that a disk drive of today conjures up, where you hold Terabyte drives in the palm of your hand. These 50MB disk drives were considered compact, as similar capacity drives of the time, were the size of washing machines. The drives the chief programmer was listening to only weighed 65 pounds (29Kg). The head positioner arm and motor were large and heavy and made a clanking sound whilst performing seek operations which were almost continuous during race meetings. When this fell silent, it was an indication that the transaction file was no longer being updated and indicated that betting had stopped.
Computers were not widespread in Queensland when this system commenced operation, it was years prior to the Personal Computer industry starting to flourish and general knowledge of computers was yet to be established. At one of the early operations during a hot period, when it became apparent that the air conditioning was struggling, the branch manager Kevin started repeatedly announcing to tote employees to "Hose it down". I followed repeatedly protesting No! No! not the water!
There was no shortage of incidents which appeared comical especially in retrospect. One of the engineering staff Charlie Barton was removing his jumper in the operations end of one of the computer vans. He removed one arm from the jumper, then swung the jumper around to remove the other arm. As the jumper passed one of the control terminals a loud crack was emitted resulting from the static discharge. After a few blank looks at each other, we discovered the zapped terminal no longer responded to key presses resulting in the application program associated with the terminal, being hung and locked in memory. This was a problem in the days of sparse memory resources, having them wasted storing a program which was not functioning. There was a problematic procedure we devised to terminate hung applications which worked under certain circumstances. It involved fiddling with bits in an operating system control block in main memory. It occurred to me afterwards that the only thing missing with the jumper episode to make it entertaining, was the words Hey Presto after the crack and a presentation of the resulting functionless terminal.
There was a large technological gap to jump, for the engineering staff who had to make the transition from the old electro mechanical system to the new computer system. This I always regarded as an impressive achievement and the magnitude of this achievement was never so apparent as on one week end when I supervised and assisted with some work on an old electro mechanical system. This system strangely enough was being installed as an interim measure to us supplying a computer system, at a new customer's track. This was my one and only experience with a live electro mechanical system. This was a relatively modern one, part of which was based on Uniselectors. One of the technical staff Charlie, mechanics as they were called then, indicated that the presence of a DC voltage at a ticket issuing machine had to be checked. As we walked to the ticket issuing machine I noticed that the only tool we had with us was a rather large screw driver my companion was carrying. I presumed he had left his meter for measuring the voltage near the machine. When we arrived the lid of the machine was lifted and the screw driver was promptly inserted in the workings resulting in an impressive spark. My immediate reaction was oh dear, that's torn it. The look of satisfaction on my companions face took several seconds for me to comprehend. The test was successful! This was the way voltages were tested on electromechanical systems! Great pains are taken to avoid shorting anything out with electronic systems.
I have seen many innovative solutions to problems, none as blunt as this one however. Sometimes applications would hang. At a meeting in the early 1980s at Ipswich one of the Race Day Control (RDC) applications hung rendering its associated console useless. Of course as Murphy's Law would have it, the problem manifested itself when the race that this RDC program was controlling had to be closed and selling was still enabled with the race running. This is a problem as someone could purchase a ticket with knowledge of which runner is likely to win or in the worst case knowing the winner. The engineer Rob Hope at this operation came under considerable pressure to stop selling on this location and race. After attempting some eloquent solutions in vain, in desperation he halted the transaction processors! Very effective, it stopped the selling on that race and every other race on every other venue and everything else to do with the tote as well. Although this sounds drastic and I have never seen any action like this before or since this event, the impact was not that great. A restart was performed after halting the processors. The recovery software checked to see if there were any races open that needed to be closed based on the current time having passed the scheduled race start time and then performed stop sell operations on any races found in this category.
The operations end of the ATL mobile computer tote
The above image shows three of the four Race Day Control terminals. The TVs above the VDUs (Video Display Units) show some of the channels of CCTV used for public display so the operators could see what they were displaying on the screens at any particular time and determine that they appeared to be working normally. The screens in the photo are displaying output from a diagnostic. Any of these VDUs could be used to control any venue however the operators used to always apply the same venue to each position so that the left VDU was used for Brisbane, the middle for Sydney and the right for Melbourne. The fourth VDU not in the picture was used for Adelaide. It was one of these VDUs that hung as mentioned in the previous paragraph. The fourth VDU is to the right and behind where I was standing to take this photo. It was this fourth VDU that Charlie's jumper zapped as described above. These VDUs could be used to run any of the tote control applications but the RDC operators just used the RDC (Race Day Control) application. It was at these terminals that Ray in the following paragraph and other RDC operators worked. Much stress was experienced at these terminals and yet, although many problems were initially identified here the responsibility ultimately landed with the technology staff as they had to find quick solutions. As is the nature of real-time computer systems, any problem produces immediate repercussions. We not only had our customer, the race clubs, rightfully demanding explanations and instant return to normal operations, but a racetrack of dissatisfied punters, who were often well lubricated, were not far from our door. My observation over the years is that, as general knowledge of computer systems has grown over the years, most punters have become a lot more tolerant and cooperative. Technologists reading this will be wondering about the unusually high VDUs in use as RDC terminals. These VDUs were manufactured by a company called Datatronics owned by ATL. They were designed with an unusual aspect ratio as a result of observing that programmers rarely used the maximum line length of standard terminals but often thought that insufficient lines could be seen on one screen requiring too much scrolling. I found these terminals very effective being able to see more lines on the screen. I recall other programmers making similar observations.
It is not possible to consider that the maintenance demands of a system that taxed the efforts of a 9 man technical team to the extreme at times could get any worse but then there was the Jinx! Ironically he was the most beloved of our managers, Bruce. A man who managed with leadership and led by example. There were some crucial elements that resulted in the success of this project and he was one of them. The staff cherished and looked forward to his visits. The hardware and software celebrated his visits by excelling themselves in their misbehaviour. These strange coincidences were established over many years, to the extent that he would eventually steer clear of the computer systems during his visits, much to the disappointment of the staff working there. One day long after his retirement I had a terrible time during a Saturday afternoon meeting. We had three major problems the most impressive being a disk drive power amplifier board in flames. After it was all under control I staggered out of the computer van and uttered the words to my 2IC One could be forgiven for thinking that. I looked up, my astonishment keeping me from completing the sentence with the words Bruce was here. I could not believe my eyes, he was at the bottom of the staircase. He had popped in to say hello and see how we were all getting on! As always, we were delighted to see him, however wished that it was not a race-day.
The Tally printers in the mobile computer tote
As previously mentioned, general knowledge of computers did not exist at the time of this system starting operation. As a result the novelty of a new computer tote on the Brisbane Metropolitan tracks attracted some media attention. We had several visitors who were interested in seeing this new on course tote era marvel. I was operating an early meeting at the Gabba Greyhounds. I was training a newly employed technician Rob, on the computer van operations. That day we were expecting a visit from Clem Jones who was interested in seeing the system in operation. Clem was the longest serving Lord Mayor of Brisbane and had a grandstand at the Gabba dedicated to him. I was looking forward to talking to him that evening about the new computer system. A problem was experienced in the Tally printer attached to the master computer system. The photo above shows these Tally printers which were just inside the semi trailer's access door. On course pool collations and dividends at that time were being entered into the TAB system for inclusion in their pools for odds displays and dividends for payment around the state. This at the time was being done over the phone and printed versions made it easier for the operators to do this via a telephone conversation. Many methods of automating this process were implemented over the years finally resulting in an Inter CPU Link with the TAB. I cannot recall the exact nature of the problem however I recall spending some time at the back of and underneath the printers in the image. When I finally emerged, having rectified the problem I said to Rob that it must be about time for Clem's visit. Rob informed me that he had been and gone and that he had a very interesting conversation with him. I was quite disappointed not having been able to meet and talk with Clem. That's what tends to happen, life passes you by when you spend your time with your head inside a computer working on hardware or have a VDU attached to your face working on software. For those who do not recall the Gabba Greyhounds, it was a Cricket Ground and Dog track prior to being a Cricket Ground and Football Stadium. Many years later, at the funeral of one of our branch manager's Kevin, I heard a story I had heard before about Clem's benevolence. The story related how Clem had purchased a house for friends of his to live in, when the family had fallen on hard times.
A computer system has a life cycle, and as the operational cycle starts a set of bugs are inevitably identified that no amount of testing seems to be capable of finding. As the operational cycle continues the incidence of these bugs reduces until a stage is reached where you think all the software problems have been attended to, bar modifications of course. Then just as you have become completely confident in this knowledge, an important application crashes after 6 years of operation. On this occasion I was amused to find that this bug had been with us from the start. A set of circumstances that took this long to eventuate.
I have always had a high regard for my mentors. I have had some excellent ones along the way especially at Channel 10 Sydney. But none have taught me more than the Engineering Manager of Automatic Totalisators Bob Plemel. He was another leader by example and another crucial element in the success formula. During the initial installation he stayed at a hotel in the city. The system demanded full attention resulting in 7 days a week work being the norm and finishing at 2AM being common. On a few occasions the team managed to make it to his hotel room prior to midnight, for a bite to eat and a beer. His ability to lean back in a chair with his hands holding a glass of beer on his chest and catch forty winks without spilling a drop was famous.
I always enjoyed applying theory wherever possible. For example, on one occasion, one of the technicians Brian, asked for assistance with a fault he was having problems with. He had localised the problem to an op amp integrator circuit. We applied some calculus and determined that a constant of integration was missing. A zener diode in the op amp circuit was supposed to provide this offset. The diode was replaced and the circuit started to function. The technician was ecstatic. From memory this was in the start-up circuit in the switched mode power supply of a Datatronics VDU. These VDUs are visible in the photo above titled The operations end of the ATL mobile computer tote.
The reverse side to the above later applied. As the number of component failures started to subside after several years, the major problem became contacts. Eventually this accounted for approximately 80 percent of all problems. The solution we devised was to disconnect all connectors and unplug all plug in components, spray the contacts with contact cleaner and reassemble. This procedure became so common that we named it Mumbo Jumbo due to the lack of any scientific analysis. A newly employed young graduate Jeff Taylor came to me one day rather dejected with the following concern. Do you mean to tell me that I have studied Thevenin, Norton, Kirchhoff, Boolean algebra, complex numbers and calculus ... and all I needed was a can of contact cleaner?
How does the axiom go? What goes around comes around! I spent a day with my eldest son Paul, who at the time was a student of these matters, we worked on propositional calculus converting logic expressions into disjunctive and conjunctive normal forms.
The Brisbane Project which was established to develop the new totaliastor systems existed in the shadow of a much larger project Sha Tin. As a result, in my opinion the Brisbane Project did not receive the resources that it should have. When the Sha Tin project failed due to inability to deliver on time, the success of the Brisbane Project became essential to the continuation of the company as all future contracts had a qualification clause stating “Dependent on the success of the Brisbane Project”. Massive overtime was worked by the handful of employees who were familiar with the Brisbane Project to ensure it successfully opened on time. This overtime, resulting in our presence on the racetracks late at night, conflicted with the lifestyle of the Racecourse Managers, which required rising early in the morning and consequently retiring early at night. The Racecourse Managers usually had guard dogs, which fortunately, whenever I saw them, were in the presence of their owners, who quelled the dogs’ evidently overwhelming desire to sink their teeth into anyone they did not know. Around 8PM the Racecourse Manager would inform us in no uncertain terms that he was finishing for the night and letting the guard dogs out. The Brisbane project provided two separate mobile totalisator systems each housed in a semi trailer. Fortunately most of the work was on the transaction processors which were inside the semis. We developed a procedure to exit the track in which we became particularly adept. We would open the large trailer door a crack to peer out to see if there was any sign of the dogs. Having established the coast was clear we would open the door and exit, close and lock the door, as rapidly as possible, keeping noise to a minimum, as the door was heavy with a heavy locking mechanism, which tended to clank and squeak. We would then dash as silently as possible to the nearest point of the outside fence. One would clamber to the top of the fence and the other would pass up the brief cases one at a time to the person at the top who would drop them on the outside of the track. The one on the ground would then clamber up on the fence as quickly as possible to minimize the opportunity for the dogs to catch him. I recall many occasions sitting on top of this fence around 2AM relieved that we had successfully run the gauntlet once more and thinking, whilst preparing to jump down on the outside of the track, that when I entered the computer industry I never envisaged this would be the method of exiting work. I have often since wondered what the present day OH&S fraternity would have had to say if confronted with this procedure. This was our only means of exiting work so late as we were not entrusted with track keys during those early years. There was no communications to the outside as telephones on course were illegal. Wives worried at home although I think eventually they became accustomed to it.
I had been working one day with one of the technicians Lawrence, on a large lamp panel indicator board. He was an excellent technician and had been employed soon after the commencement of operation to increase the number of local staff with electronics capabilities. We were on the way back to our office in his car, when we came across a race horse with a jockey riding. We were permitted to use vehicles to cross the track with the understanding that we would be very careful. Race horses are very temperamental. As the technician had been working at the race track for some time by then, I expected him to give the horse a wide berth. We were following the horse and I became increasingly concerned as we came closer and closer to the horse. I was almost speechless, when he drove right up to the rear of the horse. I was about to utter the words "back off", when he hit the horn! Luckily the horse did not mind. The nicknames we earned for ourselves that day do not bear repeating. On another occasion, with another technician Ken Turner, who was more horse wise, the opposite was true. We had to get through a gate with the car. The technician got out to open the gate and I remained in the car to drive through. A horse and rider had passed on the other side of the gate and the technician waited for them to reach a reasonable distance from the gate. He then slowly opened the gate which emitted a very faint squeak. We were instantly treated to a bucking bronco show.
There was plenty of variety in the job. Some of the technicians also drove the semi trailers and Rolls Royce generators from track to track. One of the technicians Brian Constable, was in transit with one of the tote vans. The tail shaft failed and he was unable to proceed. He pulled up outside a hotel and applied the hand brake. As the hand brake acted on the flywheel which was no longer connected to the wheels, it was ineffective. By the time he came out of the hotel after calling for assistance the vacuum in the main braking system that was holding the van had dissipated. The van had gone! An old man waving his walking stick in the air at our technician shouted Hey Sonny was that your truck I saw going down the hill; Fortunately it had not gone far and just come to rest down the hill without colliding with anything. On another occasion whilst transporting the backup generator trailer with two technicians Brian Dalton and Ron Findlay in the prime mover, one turned to the other and exclaimed Isn't that our generator passing us. A weld had failed in the coupling.
ATL computer trailer at The Gabba
We used to have a security system. Two neglected firearms wrapped in a cash bag and carried around by a staff member. I used to wonder who would be injured if they were ever used, the target or the shooter.
A technician Bob, was having a disagreement with a programmer, Dick one day. The technician shared my stature, close to the ground, the programmer was the opposite. Frustrated the technician eventually strode off, fetched a bar stool, thrust it down in front of the programmer, clambered on top and announced "There, now I can talk to you".
Another technician, Brian Riley, different to the previously mentioned Brian, was a true gentleman indeed. He took a disliking to something that was said to him by the branch manager Roger Penwarden. Obviously angered, a unique event, he searched his mind for a damning reply. The word's eventually came, the worst he could muster "Be careful or I will go and get my big boots on".
At a hotel close to one of the tracks we serviced, they sold beer on tap stored in wooden kegs. This became famous within our company and interstate visitors would often ask to be taken to "have a beer off the wood".
The Brisbane Branch became well known within the company for its maintenance expertise. It ended up performing maintenance on the DMX11s for the whole company. This was a custom built 64 multidrop line multiplexer developed by Digital's special systems section specifically for Automatic Totalisators.
It is with some sense of loss, that one sees a system that one has nurtured so intensely for a decade, enter the decommissioning and disposal part of its life cycle. The consolation is that you are too busy with the new system to think about it. In retrospect, I spent considerably more of this decade with this system than with my family; The modules that were still used at other installations were removed. The semi trailers were sold, with the computer equipment remaining to be scrapped. One CPU box complete with contents was donated to the Power House Museum in Sydney.
I sometimes look at the remaining old spare disk drives, now an anachronism. They appear to be desperately clinging to the remnants of those magnetic flux bit patterns that represented those once all important transactions, recorded the last time they operated, oblivious to the fact that they have been unimportant for decades. It reminds me of how wasteful this industry is.
There always seems to be some competition amongst project members to have been associated with the most challenging and difficult of installations. We had some tote installation veterans on this project and they seemed to agree that this was it. I have thought long about what motivation lies behind a greeting from Dale Oxtoby, a programmer who was an integral part of this project, that I had not seen for many years. He gave me a hug that lifted me off the ground. I came to understand it recently when a 20 year reunion of the project team was suggested. Despite the fact that we had all gone separate ways, one of us having realised a life long dream of becoming a farmer and another in the process of realising the same dream, there was never any question of not attending. My understanding after the reunion was that this was the "life boat" principle. We did not share any life threatening experience however we came through a traumatic experience together, and I can only liken the result to the bond that seems to exist amongst those who have shared a crisis. The reunion was held at Coffs Harbour, a central location. The furthest anyone had to travel was from Canberra. Dale's wife Pat, broke her leg prior to the reunion however this was no deterrent to attending. One of the Motel staff where we stayed two nights asked the question "If you people all used to work together, how come you get along with each other so well ?".
The next generation system in Brisbane was VAX based. With it came a new project manager Graham, a well liked chap with a healthy ego. The embodiment of this ego was his car, a V8 Calais. As a passenger in this car and a pilot I quickly recognised his driving style to be more related to flying than driving. Whilst backing out of a parking space at Eagle Farm Racecourse one day he backed into a large old unyielding Moreton Bay Fig tree. The collision demolished the plastic parts of the car involved and taught some of the metal parts about yielding. Talk about a dented ego!
We have heard a lot of lines during the lifetime of the PDP11 system many from Kevin, the branch manager at the time, "That boat's gone", "We will cross that bridge when we come to it", "You are talking again you should be listening", "I never make decisions when I'm drinking", and one I found particularly amusing from one of the engineers Ben Barton, was "This job would be great except for the horses". Ben also taught me the phrase "tin plate your backside" which he told me came from the Army. Although I had not heard this one before, I was well acquainted with the concept.
There were always a few programmers that I cast into the genus, genius.
One of these programmers Elliott Roper, developed our new generation DMX, unlike its predecessor it was a diskless PDP11 based front end processor. The program was written in assembler and executed without an operating system. For those of you familiar with the "real programmers" sayings, he did not eat quiche. It performed its own memory allocation and deallocation, queuing, interrupt servicing and prioritising. The pièce de résistance was the state machine which kept track of the protocol on the many serial lines. Many will argue that there is an element of art in software. If we could draw an analogy to literature, I would describe this piece of software as poetry in motion.
He liked using a line editor. Commands were issued for everything. To correct a spelling mistake a command would move the cursor to the line number, another command moved the cursor to the character number yet another command deleted the offending characters then a command was issued to enter insert mode and then the correct characters could be typed.
He would work with three terminals at once with the keyboards end to end forming an arc in front of him. His hands would dart across the keyboards which gave the impression of someone playing an organ more than someone writing software. He usually wore headsets listening to classical music. A bystander could always tell when the point of peak creativity had been reached. His hands would take long vertical single fingered stabs at a keyboard and you would hear the utterances "PING" and "TING". Oh I almost forgot, he always wore boots.
|To the Technologists|
Two of the most interesting maintenance areas were the phase locked loop in the disk drive read electronics, and processor internals.
The phase locked loop represented my first experience with the ECL logic family. The rest of the system was mainly TTL with some DTL in the tape drives. We later upgraded to MOS memory.
The processor internals was interesting as it usually meant working at the micro instruction level. On one occasion a processor was failing the power up sanity checks. By stepping through the micro code a faulty flip flop was discovered in the Bleg of the ALU.
Utilising Minicomputer systems in a mobile application was an unheard of concept, which had technologists looking at you in disbelief when telling them of the mobile totaliastors based on PDP11s. There were “ruggedised” versions of these computers which I believe were mainly for military applications however we were not using them. This was so unheard of at the time that this may have been another Australian first! In practice the use of these minicomputers in a mobile application was not without significant problems. Minicomputers were very hardware intensive in comparison to the Microcomputer class machines. As previously mentioned this history pre-dates home computing and it was the Microcomputer class of machine that would become the basis of the personal computer industry. At the time the major application for microcomputers was in device controllers like the M6800 microprocessor in the J22 ticket issuing machines mentioned. Also as previously mentioned these totalisator systems were cantankerous, with no shortage of hardware failures, not to do with their manufacture, but related to the mobile application and the associated vibration. An additional contributory factor was opening with a system that was in need of additional software development and debugging, as a delay was not possible at the time and it had to open now or never. Finally add to this air conditioning that struggled during the hotter summer days triggering thermal runaway failures in semiconductor devices and you end up with major problems all contributing to the anxiety, previously mentioned, of staff working with these systems.
Considerable work went into maintaining servo systems, particularly the tape drive capstan and reel motors. A lot of disk positioner servo alignments were performed. Many Unibus problems were rectified. It is ironic that after analysing so many Unibus problems, when we installed the next generation machine, VAX based central site, we received Qbus display panels. These were never used due to the increase in reliability. The display panels would only have been useful for a cursory check only however, as many problems required dynamic analysis. The large Unibus cables that extended the Unibus into the expansion boxes were always a source of problems. It was difficult to keep them tidy and out of harms way and they always seemed to twist the wrong way at the destination requiring some fancy folding.
Something else that has disappeared is the machine code diagnostic maintenance loop. Often during the analysis of a problem the need would arise to produce a set of specific recurring signals in the hardware for monitoring with a CRO. Supplied diagnostics would rarely cater for such tight, specific and continuous loops, as it is difficult to predict what type of diagnostic loops are required for repair until a failure occurs and it is close to impossible to predict what is going to fail. These supplied diagnostics would identify a failing device or module and usually indicated an area within which the failing component could be found. Sometimes the supplied diagnostics could be configured to loop on a particular test which would produce signals suitable for identifying the faulty component. When this was not the case it was time to write your own loop. As you never seemed to have access to a development system with an assembler to write the diagnostic loop with when you needed one, it was written in machine code. We ended up with many of these fault specific machine code programs. They were toggled into memory via the console. Some pieces of equipment had recurring problem areas and after having recognised these repetitions, an assembler was used to write more extensive fault specific diagnostic tests. These reduced time to repair considerably. Look at a PDP11 machine code program
Talking about machine code, of those who knew the PDP11, will they ever be able to erase instructions like 112737 000101 177566 from their memories? I doubt it!
It is interesting to compare the 16 bit word length, 124K words of main memory 4K words I/O page, 50MB disk drive and 46MB tape storage with current figures. To give some idea of the level of integration the disk controllers consisted of 5 hex height boards. The dimensions were 15.6 X 8.4 inches.
Having mentioned the 124KW of physical memory it is interesting to compare this to the common 3GB of Personal Computer main memory after the first decade of the new millennium. There are 2 bytes in a PDP11 word giving 248KB requiring 12684 PDP11/34 main memories to provide 3GB capacity. Additionally, these memories were not the Dual In-Line Memory modules you hold in between two fingers. These were circuit boards with dimensions about 15.6 X 8.4 inches which you held firmly with two hands when inserting them or removing them from the backplane. MOS memory was compact and was implemented on one board of this size however core memory consisted of two boards of this size which were joined together. This totally enclosed the core stack between the two boards. Taking the first implementation of memory in these systems which was core, the boards just described contained 16K words of memory so there were 8 of these to provide the 124K words of main memory requiring 101,472 of these boards to match the 3GB of modern memory. For anyone who finds the addressable memory of 124KW odd, there were 128K addressable locations on the Unibus however the top 4K was allocated to the I/O page where all the device control and status registers existed.
The heavy clanking disk drives mentioned in the previous section were Winchester technology disks manufactured by Okidata. They used the, innovative in the 1970s, Modified Frequency Modulation method of recording which gave them a packing density of approximately 5600 bits per inch. There were 678 data tracks per surface, 8 heads and 3 platters. Maximum seek time 75 Msec average 38 Msec. The size was 7 X 22 X 19 inches weighing 65 Lbs. These drives were revolutionary with equivalent drives at the time being considerably larger. These disk drives are visible in the photograph titled Inside one of the computer vans above, one at the top of each of the second and third racks. The black horizontal bar across each drive was an afterthought to stop the drives sliding out on their runners whilst the vans were in motion. Another afterthought was the grille that can be seen around the control buttons. One day tote betting stopped. After a fallback procedure to the Slave computer system, we discovered that the write protect button on the Master computer disk drive had been activated. This puts an end to all write operations to the drive resulting in no further bet transactions being recorded. Someone had accidentally brushed up against it and pressed it in. The grille prevented another accidental operation of this critical button.
Okidata disk drive positioner arms
As mentioned in the previous section, the clanking emanated from the head positioner arm moving between requested cylinders. The photo above shows these positioner arms complete with the moving head assemblies at the upper end of the arms and the positioner motor at the bottom end. The engineering documents underneath the assemblies shown give some idea of their size.
Having introduced the disk drives it is interesting to put these physically large and heavy by today’s standard disk drives into capacity context. The storage capacity of these drives was 50MB and the drives available nowadays in 2010 are up to 1 to 2TB making the 2TB drives almost 42,000 times the capacity of these late 1970s drives. To match this capacity with the 1970s drives you would need 41,944 old drives which would weigh 1,216,376Kg to equal the capacity of a 2TB drive that can be held in the palm of your hand. Contemplating this analogy further, I would hate to think of the power bill as the spindle motor consumed significant power particularly on startup. With this mass of old style drives the power up would have to be staggered to limit surge current. This had to be done even when you had a few of these old drives. It would probably take days to start this many drives up! It is interesting to contemplate the force applied by the spindle drive motor pulley to the spindle pulley belt. This belt looked heavy duty for the application, somewhat like a light fan belt in a car. The spindle motor acceleration on start up was so great it would stretch this belt. This belt was rigid and taught when stationary. This stretch on the pulling side of the belt, caused some slackness in the belt on the return side. This slack section of belt would often result in some vibration, which would emanate a BRRRRMMMMM sort of sound until the motor was close to synchronous speed.
The tape drives used Phase Encode recording giving a 1600 characters per inch packing density. They also were capable of Non Return To Zero Inverted recording method at 800 characters per inch for compatibility. The dimensions were 24 X 20 X 19 inches and weight 155 Lbs. Reel diameter was 10.5 inches and we used 2400 ft tapes.
In the Memories of a System Long Gone section above, I mentioned a problematic procedure we devised to terminate hung applications. It was essential to retrieve unusable main memory allocated to any application which had hung. An abort command would sometimes not run to completion leaving the application marked for abort without the abort running to completion. In this event it was time to resort to Open, a system utility program for examining and altering the contents of main memory corresponding to addresses where the operating system resided. This procedure required the use of the task builder map to locate the absolute address of the start of the system task control blocks. An offset was then added to this address to point to the beginning of the task control block of the application in question. Another offset was added which produced the address of a status word in this block. Now one bit had to be set in one byte of the word leaving the high byte in the word along with the other bits in the low byte unaltered. If you got this right the program would abort. The implications of getting it wrong were some form of erroneous system behaviour or in the worst case scenario a system crash.
The fact that the systems were mobile combined with air conditioning that did not cope well during hot periods, resulted in no piece of equipment escaping failure on multiple occasions. There was always something in need of repair. I think the most reliable pieces of equipment were the KSR43 Teletypes. These are visible in the photo above labeled The PDP11 tote Console Terminals. When thinking of reliability an often overlooked piece of equipment is the test equipment. Our Tektronix 465 Oscilloscopes purchased in 1978 are still our prime piece of test equipment today 11 February 1998.
A significant improvement in the departments capabilities was realised after many years of operation, when we had appropriated sufficient parts to put together an additional simplex system. This became our maintenance system and for the first time maintenance could be performed without the fear of taking a retrograde step with one of the systems required for operations. This system went on to serve a second purpose. It allowed us to establish a development system and perform our own software maintenance. The software was written in Ratfor a structured version of Fortran. Once the software maintenance was established we had need of a tape drive for backup, as the disk attached to this system could be off to see active service any time one of the disks in the on line systems failed. We then needed to be able to restore the latest copy of the development system to the new disk once it was repaired.
We managed to acquire a TU10 tape drive and controller from the Sydney head office. This was a faulty drive that no one else wanted. It had not been used for a long time. It was a museum piece then. There are two types of faults that are more difficult to repair than others. These are compounded faults and intermittent faults and this drive had both. By the time we had it working there were 11 individual repairs logged in the maintenance log book. The first job was to acquire a vacuum motor, which was missing on the TU10. Another fault provided an opportunity to apply some of the theory mentioned above. The tape was moving at 61 inches per second in synchronous forward instead of 45, with the forward speed servo pot adjusted to minimum velocity. The output of the capstan tachometer was insufficient to bring the capstan motor speed within adjustable limits. As we could not acquire a replacement tachometer an amplifier was designed and installed to provide the correct level to the capstan servo master summing amplifier. After contending with the 11 faults we had our backup. I cannot recall any other occasion in my maintenance career when a piece of equipment had any more than 2 individual faults let alone 11!
When a failure occurred resulting in the need for a restart many transactions could be stored in the shared memory linked list that were not stored on disk. A program was quickly written to flush these transactions to disk. This however sometimes left the transaction files on both disks which were supposed to be identical, differing from each other. It was then necessary to manually cross reference the transactions in question to ensure that they were all recorded on both disk drives. These sorts of problems received top priority and were rectified as quickly as possible.
In the decade of operation, reliability improvements were achieved, however one never felt confident that you would conduct an operation and have a trouble free day. These improvements resulted from three events. First, the computer systems in both semi trailers were reinstalled, paying particular attention to cleaning up the mass of wiring which lay in a fairly hostile environment with many jagged edges inside the racks. Secondly, improving the air conditioning system. Finally the burn-in principle. I became a firm believer that systems underwent a burn-in period. Each time the system was subjected to higher temperatures the marginal components would fail. The more components failed and were replaced, the fewer marginal components remained improving the reliability. Each time we had air conditioning problems with the predictable component failures, the more robust the system became after the repairs.
One last note on maintenance. It has nothing to do with PDP11s however during the lifetime of this system I was sent to Boston to attend a course on VAX11/750 processor internals. These machines were being used elsewhere in the company and needed some in-house support. A language had been developed for writing micro code for this processor and we were examining parts of this, at the course, to familiarise ourselves with it. I knew the architecture of the VAX and when the lecturer started talking about writing to a particular internal processor register I asked the question "That is a read only register, why are we writing to it here?". The instant the words had left my lips I knew what he was going to say and wished I had thought about it a few seconds longer before speaking. The answer was quite obvious, "We are implementing that architecture with this code and the processor has to be able to write to this register at the micro code level so that the macro level machine code instructions have something meaningful to read from it". This was an epiphany for me and I instantly felt that I had walked through a proverbial door. I was no longer on the outside at the assembly programmer's terminal looking in at the architecture, I was on the inside with the computer architect implementing the architecture, looking out at the assembly programmer!
To complete the chronology of the on course tote systems in Brisbane I will make a quick observation of the VAX system and its successor. The VAX was a central site system. Work commenced on the Brisbane VAX tote project in 1986. A significant level of improved reliability was attained from not having to move the system from track to track. The job went from being hardware intensive to being software intensive. The Brisbane branch performed its own software maintenance and enhancement programming. This was done using Pascal and VAX assembler. We also assisted the Systems Department with overseas projects when their resources were being over committed.
The ATL VAX based totalisator system
Similar to the PDP11 system this was a duplex system operating in hot standby. The product name for this tote was Atlas 2000. This was a fixed site installation at Eagle Farm Racecourse which catered for the other tracks at Doomben, Albion Park Bundamba via two modem links each. The Gabba Greyhound operations moved to Albion Park and operated on different nights to the Trots. The first two racks contain the on-line master and slave systems. Which system operated in which role was selected on tote system startup. The third rack was a huge improvement to the predecessor system. This was a complete spare third system right from the start. It was available to provide quick module repair for the on-line systems and was used as a maintenance system on which the failed module could be repaired to component level. This third system was also a complete development system so that the Brisbane department could perform all bug-fixes and enhancement programming in house. At the bottom of the first 3 racks are DEC MicroVAX IIs. The two black devices half way up the first two racks are DMXs. These are ATL developed front end processors based on PDP11s. This is the diskless PDP11 based DMX developed by Elliott Roper as mentioned previously. The third rack also has a DMX however the front panel has been removed for maintenance work purposes. The far rack contains the communications equipment. At the top of this rack there is a large modem which utilised Telecom's ADS service, (Analogue Data Service not to be confused with the much more modern ADSL (Asymmetric Digital Subscriber Line)) to provide the inter CPU link with the TAB system at their headquarters at Albion. This service provided lines which did not go through the PSTN (Public Switched Telephone Network), and were a permanent end to end connection. Below these are three shelves with two Codex 2260 modems on each shelf to connect to our other tracks we serviced as previously mentioned. Below these there are two boxes which look like they have two horizontal white lines across the front panels. These were PDP11s based on the J11 IC. They were used to provide protocol conversion between our asynchronous serial communications and the TAB's HDLC based synchronous communications. This was part of the inter CPU link with the TAB. Not clear in the photograph, on the next three lower shelves are a modem and two JNA Statistical Multiplexers used for operating the Sunchine Coast Turf Club at Caloundra. At the bottom of the racks is a Trailblazer modem. These systems were left running 24/7 and I used this trailblazer to connect to the system so I could access the system to provide technical support in the event of a problem during a shift for which I was rostered off. Additionally I could do urgent programming work from home after work or on days off. This sort of thing is commonplace today but was a great innovation at the time. Additionally, the ATL operations branches did tend to lag behind the systems department in Sydney when it came to uptake of new technologies. At the top of the first two racks are the I/O patch panels. These connect DMX comms lines to the TIM (Ticket Issuing Machine) lines. Behind these panels are level converters which convert the RS232 used by the DMXs to RS485 used by the TIMs. The RS485 is used for the long runs to the tote houses where the TIMs are located. For those unfamiliar with RS485, it is a differential line standard providing a high common mode rejection ratio for noise immunity for transmission over longer lines. The lines on these patch panels are organised in pairs A and B. Each bank of machines on a TIM line was supported by two comms connections A and B. All the A connections go to the left hand DMX and all the B connections go to the right hand DMX. The TIMs were dual ported so the A line connected to the first port and the B line connected to the other port. In the event that there was any problem with one of the communication paths or a DMX failure, the other path or DMX took the full traffic load. The ATL Digital Data Communications Protocol supported 16 TIMs per line. Frelco connectors were used on the patch panels. This patch panel was a good starting point for looking into any line problems using a Line Monitor and a CRO. This is the same arrangement with the predecessor system. Looking at the photo above titled PDP11 systems inside one of the computer vans these I/O patch panels are concealed behind the double doors at the bottom of the last rack in the photo. The TIMs used with this system were mainly J25s. There were some J22s and J36s.
VAX/VMS Operating System Manuals in my office
As mentioned above, the Brisbane department performed all the maintenance and enhancement programming in house. Above is an image of the VAX/VMS Operating system documentation set. This leads to an observation of a significant difference between the Mini Computer era and the advent of the personal computer industry. This documentation set provided information on every facility of the operating system, including all the system services and every piece of functionality and every command and every argument of every utility programme. It also provided total documentation for all of the software development tools. In my experience everything written in these manuals was exactly how these machines worked and you could become an expert by reading them. There was never ever any need for guesswork. When the Personal Computer industry burst into being I observed a generation of budding technologists who were accustomed to an industry where experimentation was a prerequisite to understanding functionality. The Mini Computer Manufacturers catered to a professional client base who knew exactly what was required and what could be expected of their systems. The Personal Computer business brought computer general knowledge to the masses so initially, the primary clientèle were novices who by nature were more tolerant and accepting of the notion that there is some fundamental law of physics that they were yet to discover that dictated that the way it is, is the way it has to be! The deficit of comprehensive documentation in this industry left a hole which afforded many a third party to write a book to fill in the gaps. The wonderful documentation of software in the Minicomputer Industry, applied to hardware as well with engineering drawings showing every single component in the system and their interconnection as well as technical manuals describing the workings of every circuit in every module and their interaction. What a wonderful experience it was working with these systems!
Having mentioned maintenance and enhancement programming I will write a little about this. There were many facets to this software maintenance and enhancement. The most prevalent work was done on the transaction processing system's software. In the PDP11 era this was implemented using Ratfor (Rational Fortran), Fortran and PDP11 Assembler and in the VAX era, Pascal and VAX assembler. PDP11 assembler was still used in the VAX era for some of the firmware. As an example of the transaction processor software maintenance, following are a couple of entries from the Brisbane Bug Book. A lot of the software enhancement changes are quite routine for example improvements to public TV displays of odds or dividends often requested by punters who will make comments like I will frequent you tracks more often if this enhancement is made to your displays. Government inspectors would often request changes for additional information or the way it is presented in reports. For the first example I have chosen something that is anything but routine. It is a crash of "Reply" because it is one of the critical applications and the repercussion of this fault is extreme. Reply performs memory resident database updates from transactions in the transaction file hence without it, most tote functionality ceases hence this fix is a top priority job.
BBB1012 During session 287 Reply terminated due to a subrange assignment value out of range in process__signon__record. Teller also terminated due to the same error in teller__program. Reply and teller have terminated on several other occasions due to the same fault. TIMs have also been signed on with incorrect teller IDs and passwords. See software reports Atlas 28, Atlas 12 and Atlas 2 which relate to other manifestations of this problem.
The process__input procedure in teller calls the handle__signon__req procedure passing two arguments "id" and "password" by reference. These are variables within a bqueues buffer. The handle__signon__req routine deallocates this buffer prior to manipulating these variables. If teller is pre-empted after the buffer is deallocated, and before the variables are accessed, then these variables can be changed by the pre-empting process and teller accesses these corrupted variables when teller is rescheduled.
Ready for test 27/Feb/89
Tested 2/Mar/89 implemented 2/Mar/89 moved to bugs directory 2/Mar/89
*************** Tape sent to Sydney 03 March 89 ***************
The BBB prefix in BBB1012 above is a mnemonic that stands for Brisbane Bug Book and the following digits are the number of the entry in the book. As indicated at the end of the Bug Book entry, tapes containing the bugfix and associated documentation was sent to head office in Sydney so they have a complete copy of the latest software source for this project.
The second bug book entry example, BBB1049 was selected as it demonstrates a recurrent problem when you have a link with an external organisation. When you have an inter CPU link with another organisation you are not privy to the details of all the inevitable enhancement upgrades that are made at the other end. Sometimes something is changed at the other end that does not present any evidence of a problem at the other end and that despite testing with your end, causes you grief at your end when on-line.
BBB1049 12 June 91. At the Eagle Farm meeting on 10 June 1990 Tabin crashed with an array index value out of range error. Tablink was down from 12:50 to the end of the meeting. All attempts to get tablink running again failed. On attempting to call zap in the debugger a pointer reference to null was logged and dumping the tablink received messages after the meeting revealed that the attempts to patch the faulty transactions with bytran was doomed by the quantity of transactions between the faulty ones.
On examining the dump after the meeting found the TAB had sent us off course notification messages with a runner number of zero which was used to index the actual runner map in the cards database to validate the runner where tablink fell over. Implemented a check for a zero runner number in process__off__course__msg in tabin.pas.
Implemented both systems June 91
Moved to bugs directory 3 October 91
It is this sort of event that led me to be very cautious when I was writing communications link software, checking every parameter received I could, for being within reasonable limits. Working in operations I had seen too many instances of communications software crashing as a result of changes at the other end. My checking of received parameters in my code must have been considered excessive as it led the chief programmer who had been looking at some of my code to make the comment "Do you have nightmares about these sorts of possibilities?" The question took me by surprise however the answer had to be Yes. The motivation behind avoiding these problems wherever possible is heightened when you are the one who has to explain to your customer any deviation from normal operation especially when the penalty for it being your company's fault could be financial compensation.
It was good for the operation, having a development system on course. A debugger was used to rectify several problems by manipulating data or control structures on line to get through a meeting prior to a bugfix being implemented. In some instances were the problem was fairly obvious, a new bugfix could be implemented during a meeting to rectify some errant behaviour of an application.
As examples of the diversity of software being maintained, changes were made to an Okidata disk driver to improve handling of disk seek errors and debugging and fixing a problem with the J11 chip based PDP11 protocol converters. The modification to the Okidata disk driver, after having analysed system crash dumps, involved increasing the number of retry attempts and eventually resorting to a recalibrate command, to attempt to recover data from a disk encountering servo problems. This modification averted a system crash on multiple occasions, keeping the disk drive functioning long enough to make it through a meeting after which maintenance work could be performed on it. The J11 chip based PDP11 protocol converters mentioned were used for converting TAB synchronous HDLC protocol to the on course asynchronous communications. The Win Pool Grand Totals intermittently displayed erroneous massive investments. When the next odds message was received this would be replaced by the real figure. This was a significant problem as it had the propensity to encourage punters to wager more than they would have otherwise unless they realised this was an erroneous figure. It was a high priority to rectify this problem. I determined that we were not receiving any odds messages with inflated win pool GTs. Suspicion fell on our protocol converters. I had a look at the assembler source for the firmware in this device and could not see any obvious problem. The incoming messages from the TAB were being stored in a forward linked closed loop linked list. I implemented some debug code to report on parameters associated with this list. To keep a long story short, it became evident that under normal operations the linked list never wrapped around. The wrap around feature was only used if the reading procedure did not catch up with the writing procedure before the writing procedure got to the end of the buffer. It became apparent that these corrupted win pool grand totals only occurred when a wrap around occurred getting to the end of the buffer and starting at the beginning overwriting already processed messages. This caused me to scrutinise the software that manages the wrap around. It was found that the writing procedure would overflow the end of the buffer by two bytes before wrapping around. The two extra bytes overwrote a word allocated to a variable used to store the Win Pool GT. The firmware source was changed to wrap around at the real buffer limit and the problem was solved.
Having mentioned variety in software, I was sent on an urgent course in San Diego on an ICP (Intelligent Communications Processor) which would be used to implement a synchronous version of our front end processor which to date was only available in asynchronous versions. I had an event that tickled my funny bone on my return flight to Australia. I was booked on a late night flight out of San Diego to San Francisco for the first leg. The departure time was around 11PM. When I went through security scanning at the airport I came across a procedure that I was not accustomed to. Walking through the scanner the alarm was activated. In Australia the security officers would then resort to the use of a hand held scanner to identify the location of the offending object. In San Diego I was asked to divest myself of something and then walk through the scanner again. Remove keys, walk through, BEEP. Remove glasses, walk through, BEEP. Remove belt, walk through, BEEP. Remove coins, walk through BEEP. Empty pockets, walk through, BEEP. Remove shoes, walk through, BEEP. An attractive young African woman was conducting the operation and after I started to find this procedure providing a rather comical spectacle, I made the comment to her that if we keep this up we could be here all night. Her reply was quite unexpected - Oh don't you worry about that, we will get you down to your underpants if need be. Fortunately that was not necessary and it turned out to be the light metallic cover on a concertina address book I kept inside the cover of my diary in my shirt pocket.
The strangest software job I worked on was for the Greece project. This project was being developed for a target system of Alpha Servers which were an upgrade path for the VAX. This involved translating all the VAX assembler parts of the tote software to Pascal as the Alphas had a different hardware architecture. The Pascal translation was used to build executable images to replace the images containing VAX assembler parts. The chief programmer mentioned we were short of available programmers who were comfortable with VAX architecture and hence VAX assembler. I mentioned that as I had been repairing VAXs for some time I was well versed in VAX architecture and as I had been performing Pascal maintenance programming for some time I may be able to find some time to lend some assistance. I had not learnt to keep my mouth shut! I was immediately recruited to a far greater extent than I had envisaged. I found this translating from one language to another quite curious. As, like any assembler the source code is intrinsically linked to the machines architecture and it made translation a lot easier to define variables that related to the VAX architecture, wherever possible like R1 R2 R3 ... PSW PSL SP etc. Of course any VAX hardware functionality altering register contents also had to be emulated in Pascal. I often wondered what some future programmer maintaining this code, who did not have an assembler background, would make of these strangely named variables. Later there were programs created to translate VAX assembler to a language from which an Alpha executable could be built but this was after we had done this manual translation. We got the job done and my assistance was much appreciated. I spent significant time on this project whilst on holiday in Tasmania.
The VAX based totalisator Operations Room
The above photo was taken in the VAX operations room which was next door to the computer room in the photo above this one. As with the PDP11 system above this is where the Raceday Controllers worked. This photo was taken during a live operation, the occasions being Gloria's retirement and this was her last meeting. Her shift was over and she was preparing to depart for the last time. Unfortunately she has her back to the photo. The others are Dawn, Lorraine I think, Fay and Ray. I had worked alongside all but Lorraine for a very long time and with Gloria and Dawn for something close to a quarter of a century.
We installed a new system in 2003 whilst we were part of TAB Limited, which I had not previously worked on. I found it very interesting to see how differently a system could be designed. This system had many benefits resulting from innovative use of off-the-shelf technologies. These technologies were not available when the previous systems I had worked on were developed and consequently had to have everything designed from the ground up. With the new system, transaction processors, ticket issuing machines and display systems were all PCs. The new system's software was not constrained by the limited main memory resources which required efficiency to be considered with the previous systems. The faster more modern processors meant that more use could be made of script language which had been kept to a minimum in previous systems due to issues like the run time symbol translations affecting performance. Spanning a quarter of a century we have gone from a technical staff of up to 10 full time down to 2 full-time staff. I have a good friend Brian. He is a check and training captain on Boeing 747s with Qantas. When I asked him what he thought of the "new" 400 series, shortly after his conversion from "the classic" he replied that the operation was far superior. This was despite the loss of two crew members to technology, the flight engineer and the second officer. I feel exactly the same way about our situation. Ironically I have never felt so "at ease" with any of the predecessor systems as I do with this one. The two staff remaining are myself, Brian Conlon, and you guessed it, another Brian, Brian Riley. As a result of the TAB Limited takeover, we ended up operating additional tracks, Redcliffe, Gatton, Rocklea and Beaudesert. TAB Limited was later taken over by Tabcorp.
I am going to allow myself the liberty of diverting here. Some moments, unknown at the time, become life time memories. This one has nothing to do with totalisators except that it is of a technical nature, aviation, and George Julius presided over the establishment of a Division of Aeronautics along with other interest as described in the last two paragraphs of the The Shaft Adder in the Image chapter of this website. Another connection is described in the next paragraph. Yet another Brian, mentioned in the last paragraph, taught me a major part of what I know of aviation. In January 1972 during my ab initio training, prior to first solo, we were on climb out of Bankstown heading for the training area. I noticed Brian had craned his head forward and seemed to be intent on one of the instruments. I scanned the instruments in the vicinity of his fixation and could not find anything wrong. I looked at Brian to check the vicinity of the instrument panel that had captured his attention. Again I could not find anything wrong. I thought that any second now I would be informed of what I had missed. When no words ensued I moved my head forward and looked back at his face. His eyes were closed! The words that eventually came related how hard the night before had been. I probably made more of this event than was intended however it left me feeling that I had been paid the greatest complement of all, he felt safe enough to have left me in command, for what was probably, albeit, a fleeting moment.
There is one more connection between the previous two paragraphs and totalisator history I will mention. The Boeing 747s that Brian was flying would have spent time in the Boeing 747 hangar at Kingsford Smith Airport. This hangar won an IES (Illuminating Engineering Society) Industrial Lighting Award for Julius Poole & Gibson and architects Stafford Moor & Farrington in 1973. George Julius' company Julius Poole & Gibson, were also involved in the design of the operating mechanisms for the enormous hangar door.
|Previous page||Go to the index||Top of the page||Next page|