Copyright © 1997 - 2015 Email - firstname.lastname@example.org
In Brisbane, my domain, these devices and the other tote facilities were replaced by PDP11 based computer totes which I worked on. The application which replaced these hardware Raceday Control Consoles was named, you guessed it, the RDC for Raceday Control Console. When the PDP11 system was replaced by a VAX based system this application ended up called the TCC or Tote Control Console. In Brisbane this transition from electromechanical to computers took place later than many other tracks which first transitioned to a PDP8 computer based system.
I have included the following information from Rob Stone who worked on the PDP8 installations at Harold Park and Wentworth Park, which in the case of the former was nine years prior to the transition in Queensland, which I was a part of. The world's first of these electronic totes was developed by ATL for the New York Racing Association at Aqueduct. To read about this system, have a look at Aqueduct in the Automatic Totalisators in America chapter. It seems appropriate to present it here, as Rob mentions the Tote Control Console. Unlike the systems in Brisbane, Harold Park had an intermediate step prior to implementing PDP11 based totes and that was a PDP8 based tote system, the subject of Rob's anecdote. The Tote Control Console Rob is referring to, is not yet a pure software application but is a custom built console that performs the function of the Tote Control Console in the image and more, which Rob describes below. There is a direct link between Rob's information and the image above as the Tote Control Console he is referring to actually replaced the TCC at Harold Park shown in this image. My short time at Harold Park, starting with an introduction to Totalisator Operations and later repairing difficult to identify faults, was associated with the PDP11 computer tote that replaced the PDP8 system Rob is writing about. Another connection between the Julius Tote and Rob's PDP8 system, is that this computer based totalisator unlike its successor, spent more time superseding Julius Totes. As a consequence, an electronic device called a scanner was developed to interface the PDP8 systems with Julius Tote electromechanical TIMs. These electronic scanners replaced the Julius Tote front end system of the same name and interfaced to the PDP8s instead of the electromechanical shaft adders. Hence this system at Harold Park was capable of interfacing to the old J8 TIMs (Ticket Issuing Machines). I do not know if this feature was used at Harold Park. I have read that 114 new J11 TIMs were part of this new system. The J11s were specially developed for use with minicomputer systems. There were some J15 included in this project as well. The new system operated on Win Place Quinella and Doubles pools. To read about the J11 see A J11 TIM which is one of the images in the Photo Gallery Continued chapter
Rob's Indicator Chaos anecdote, is very interesting to me, as it illustrates a basic principle regarding testing software before it goes live. This principle is that no amount of software testing, will identify all software bugs. When performing rigorous software testing, which is essential, there comes a time when you go beyond the law of diminishing returns. Once this point is passed you start dreaming up nightmare scenarios that will never bite you in operations incurring unfruitful expenditure. In other words there is a practical time when you have to go live with a new computer system to see what real problems remain. I am convinced that many computer systems retire with undiscovered bugs as a result of a scenario envisaged and catered for in the software design never eventuated in the lifespan of the system. The problem Rob relates, interestingly enough, is one that I regard as one of those nightmare scenarios that would never have been dreamt up during the testing phase. The ultimate success of a system depends on how the problems revealed after going live are dealt with. It is the sort of action that Rob took, that resulted in the problem being rectified quickly and that, is essential for the ultimate success of a system. Oh an a bit of luck is very helpful as Rob points out as his problem manifested itself at the end of a race meeting. Finally, he makes a reference to a well known totalisator operations problem, that if punters cannot see the odds they don't bet.
I worked for ATL for 3 years, from Jan. 1970 to late 1972, initially as a Tote System Installation Engineer under Peter Rolls, and later in the Research Dept under George Klemmer.
I was given responsibility for installing the computer systems at Harold Park and Wentworth Park, supported by Peter Rolls, Ron Hood, Kevin Franks etc. in Installation, George Klemmer and team in Research, Dick Sterndale-Smith and programming team under George, Neville Mitchell, Alf Lesins and team. in the Drawing Office, and Alan Lakeman, Bill Johnson and team on track.
After Harold Park was up and running, as far as I can remember, there were very few operational dramas, except for one memorable Saturday night.
The dog meeting was going smoothly, right up until 5 minutes before the last race (Race 10). This was when the TAB off-course betting data was entered into the on-course pools before each race, by an operator using an old ASR-33 teletypewriter (you remember them, don't you?).
Suddenly, absolute chaos descended as all the odds display boards on course "went crazy". Random draft dividends were displayed as the updates rippled down each column on the boards, then were overwritten as it went back to the start immediately and clobbered that data with further random dividends. This is all just as the majority of punters were waiting to see the changed odds after the off-course betting was added, before placing their final bets.
Very soon, Alan Lakeman and co. burst into the computer room, naturally demanding "Rob, what's going on???". I had already had time to take the first course of action, which was to take the B computer off-line, see no effect, put it back on-line, then take the A computer off-line, and (unfortunately) see no effect there either.
Of course, everyone wanted to know "How can the punters place their bets if they can't see the odds?" Unfortunately, the answer was: Either they have to go on the last displayed odds, or sorry, they don't bet.
As both computers were issuing chaotic but identical behaviour, I'd had to conclude that it was a software problem, and told Alan this. I said that therefore there was nothing else that could be done at the time, but just to keep our fingers crossed and hope that that the betting data stored in memory was safe (no disks in those days), and that the computers would stop accepting bets when the race started, and could then calculate the right dividends when needed. Fortunately (phew!), it was the last race of the night, everything else except the odds indicator boards appeared to function OK, and the race and night completed successfully.
Also fortunately, I had been able to observe exactly what had happened just prior to the calamity, and reported the details (I think with a printed memory dump), to Dick S-S first thing Monday morning, who was straight onto it. In entering the off-course betting info on the ASR-33, the operator had made a mistake, and had gone into a 'correct' or 'edit' mode, using 'Backspace'(s) etc. Unfortunately while in this mode, she made another error (probably under the pressure of hurrying before betting closed), and tried to correct that. And that's when "all hell done broke loose, boy".
In all the software testing that had been done, that was one combination of events that hadn't been anticipated, and the program tried to store part of the double-corrected data in the area of memory where the indicator board update program code resided - thus sending that software routine haywire. Also fortunately, that routine was the last in the memory map, and so other program code had not been affected.
Happy ending - The problem was fixed promptly by our software colleagues, and I don't recall any other significant incidents during the commissioning and hand-over periods at both tracks.
Webmaster's note: Rob's crippling fault caused by a software routine overwriting memory dedicated to storing instructions, reminds me of a similar problem in Brisbane. It was a bug overwriting a critical variable storing the Win Pool Grand Totals, resulting in intermittently displayed erroneous massive investments, on CCTV and Indicator displays around the track. I remember it well as it was difficult to isolate, as it was not possible to reproduce the symptoms in a test environment. We had J11 chip based PDP11s acting as protocol converters, converting TAB synchronous HDLC protocol, to the on course asynchronous digital communications protocol. This link sent the on course pool figures to the TAB and the TAB returned the figures for the whole of Queensland. This was a significant problem, as it had the propensity to encourage punters to wager much more than they would have otherwise, unless they realised these massive investment figures on display were erroneous. Unlike Rob's condition, this problem manifest itself on multiple occasions and at the most undesirable times, only at big meetings. As it was not possible to recreate the symptoms, the only means of resolving the problem was to run versions of the application with debug code I had written to report pertinent parameters whilst running live at these big meetings. I determined that we were not receiving any odds messages with inflated win pool GTs from the TAB. The incoming messages from the TAB were being stored in a forward linked closed loop linked list. I implemented some debug code to report on parameters associated with this list. Analysing debug information it became obvious that under normal operations the linked list never wrapped around. The wrap around feature was only used if the reading procedure did not catch up with the writing procedure before the writing procedure got to the end of the buffer. Furthermore the corrupted win pool grand totals only occurred when a wrap around occurred after getting to the end of the buffer and starting at the beginning overwriting already processed messages. Consequently I scrutinised the software that manages the wrap around. I found that the writing procedure would overflow the end of the buffer by two bytes before wrapping around. The two extra bytes overwrote a word allocated to the variable used to store the Win Pool GT. The firmware source was changed to wrap around at the real buffer limit and the problem was solved.
And just reflecting on (gentleman) Alan Lakeman at Harold Park, I recalled a conversation with him that will stay with me forever:
Rob (reporting an equipment problem): "It's not working".
Alan: "What do you mean it's not working?"
Rob: "It's cactus f**ktus".
Alan (feigning shock & horror): "Rob! I'll have you know that I'm a lay preacher!"
I'll always remember the look of mock horror and indignation on Alan's face, before we both fell about laughing ... A memorable incident for me.
Rob Stone with Duplex PDP8 system at Harold Park 1970
The PDP8 based totalisator system shown above was Australia's first computer based totalisator system. Furthermore it was the first computer based totalisator system in the Southern Hemisphere.
(Webmaster's note: I love this image! Although I started with Automatic Totalisators on the following generation of computer totes based on the PDP11, this makes me feel completely at home. The computer consoles, the register lights, the paper-tape readers, the workbench, the paper-tape programs, the notes, the Biro, the documentation, the reel of wirewrap wire, the screwdrivers, the solder sucker, the Tektronix CRO (Cathode Ray Oscilloscope) on the floor, a model I have spent much of my life attached to, it could be me in this image! Home Sweet Home! In all the staged publicity and sales photos, it is all clinically clean, however this image shows it the way it was for the engineering staff and represents the workplace that I spent far too much of my life in! )
The photo above was a real 'on-the-job' working snapshot in time. As Brian mentioned above, all of the tools visible in the photo - from the small spool of wirewrap wire, to the Tektronix 'scope almost hidden from view, indicate that it was probably in Nov. or Dec. 1970, during the installation and testing phase, before Harold Park went live in Dec. that year. The area I'm working on was a wire-wrapped panel which held the DEC discrete component R-series and W-series logic cards that interfaced the computers into the 'twin' configuration, and provided the latching and buffering of data from the computers, for handling by the other cabinets in the system. The other cabinets as I recall were, from the left: 1) The "Interface", which defined and handled which computers were on-line, and interfaced to the TCC (Tote Control Console) and the following equipment; 2) Two electronic Scanners, each of which was a cabinet of R- and W- logic cards (in a wirewrap panel) that scanned 64 TIMs, to see if any had grounded their 'bid' wire, then controlled them; 3) The BCU (indicator Board Control Unit), which had logic boards for latching data and selecting the lampboxes to be updated next, and also the relays that released the latching voltage from each lampbox group in turn (which dropped out the lampbox relays and hence their displayed digits), before a digit code was applied to each selected lampbox, if it was not to be blank.
When being updated, each lampbox would receive a "2 out of 5" code on 5 parallel wires, which then displayed one of the 10 digits, 0 to 9. The "2 out of 5" code was a safety feature so that if a fault occurred activating (say) 1 or 3 of the 5 data lines, an irregular light pattern would be displayed (indicating an error), rather than displaying the wrong odds digit.
The TCC at that time was the physical input device that controlled the tote operation, after the programs were started in both computers. It was a neat, smart-looking desk-top panel, on about a 20 degree slope towards the front, fitted into a shallow housing about 2 ft wide, by say 15 - 18 inches deep, a few inches high at the front and about 6-7 in. high at the back. It had the full complement of illuminated push-buttons to indicate Race No., non-scratched starters, which pools were active, 'Start' and 'Stop' betting, 'Calculate Divs' when the results were official, etc. It sat on the bench opposite the computer system, next to the wide computer room window, where the operator could see race starts, 'Results Official' type indicators, etc
When a TIM locked in a bet request, it would ground its bid wire (1 for each machine), seeking access to the Scanner mentioned above. The Scanner would cycle around its 64 TIMs in turn (at the rate of about 5000 checks per second), and when it found a 'bid', it would stop on that TIM and provide an enabling voltage to the TIM, which gated its bet data on to 3 or 4 groups of 8 parallel wires (no serial communications at ATL at that stage), using a "3 out of 8" code for each significant TIM figure. The Scanner would then interrupt whichever computers were on-line and present the data for processing by the computers. Each computer would separately analyse the data bid from the TIM, and issue an 'Accept' or 'Reject' independently to the Scanner. If the responses agreed, the Scanner would issue either a 'Confirm' to the TIM and the ticket would be printed, or a 'Reject' and the TIM keys would be released with no ticket printed.
If the responses disagreed, or if no response was received from 1 computer within the time allowed, the Scanner would freeze, with bid data and the 2 responses displayed for the engineer to analyse. At which point, 64 TIMs were being held up, and the engineer had to quickly look at the bid data, assess whether it was a valid bet or not, whether betting was still current on that race or not, decide which computer was right, and take the other computer off-line with the flick of a switch on the Interface. That allowed the ticket to be accepted or rejected by just one computer, and freed the Scanner to move on to checking the other 63 TIMs.
As the computer processing time for each ticket was a small fraction of the time taken to actually print a ticket, and because each Scanner could move on to the next TIM as soon as it allowed a TIM to latch the Accept or Reject response, from memory, a Scanner could issue responses to all 64 connected TIMs within 1 to 2 seconds, even if all 64 had locked in a bet bid at the same virtually the same time.
On a software note, each computer in the duplex or twin system ran independently, but each, set and cleared a flag which was incorporated into a synchronisation loop. My understanding of the mainline logic was that it consisted of a list of checks for inputs from all of the connected system equipment described above, as well as housekeeping tasks, (recalculating odds, printing betting and dividends information after a race, etc), plus the setting, checking and clearing of the sync flags. The loop was used as it was very important that each computer saw the same information at exactly the same time (or as closely as was possible), so that there wasn't a situation where the computers were processing bids from 2 different Scanners at the same time (as both would lock up waiting for the other computer), or where one computer could process a bet just before the "Stop Betting" signal while the other processed the same bet just after the "Stop". So at the start of the sync loop, each computer would set its sync flag, then wait (probably a very short time) for it to see from the Interface that the other computer had set its flag. At that point, each would clear its own flag and move off in 'lock-step' down the mainline program code, processing inputs and outputs as it went, until it came back to the 'top' of the sync loop, to repeat the process again.
However, I have digressed. Back to the photo ....
As wiring or design faults were found in our testing, it was necessary for the Engineer (yours truly in this case) to work out the change needed, remove the incorrect wire-wrapped connections, put the new ones in using a wirewrap gun, and re-test to see if that had fixed the problem - as well as checking that I hadn't added a new fault.
Seeing the complement of tools and equipment in the image, my guess is that it the photo has caught a "technical field change" in progress at the time it was taken. If a change merely corrected a wiring error, where the connections didn't match the wiring diagrams (normally 20 to 36 x A3 pages per unit), it was just a matter of fixing the connections. If the wiring had been correct, but the logic needed changing to achieve the objective, I'd also mark up a copy of the wiring page(s) and send it(them) back to Neville in the Drawing Office to be updated and re-issued.
While integrating and testing the Harold Park system in the front test room at Meadowbank, before it was moved to H.P., a highly unusual printing problem turned up.
The symptoms were (are you ready for it?): All printouts from the tote software would print reliably on the DEC line printer (80 columns - I think it was imaginatively labelled as an 'LP-80'), except for when a prior page printed on less than half the page, and was followed by a 'Form Feed' command (skip to the start of the next page), the first line of print on the next page would be completely scrambled to any random characters that the printer could produce!
OK, before you see the reason, bets are being taken on the final cause and solution.
As it only ever occurred after a Form Feed operation, the tote software was checked first. Nope - even a simple custom diagnostic program could reproduce the problem. (Webmaster's note: Oh dear - Rob has compelled me yet again to sing my accord! Any minicomputer hardware technologist at the time, worth his salt, was very well acquainted with the architecture of the machine. It was a common requirement of the fault finding process to be able to write short machine code diagnostics which were loaded into memory using the console and executed to provide information on a particular problem.)
Old reliable Tektronix 'scope to the rescue. Fault finding in the printer logic revealed that when it was doing a Form Feed, AND when the page drive mechanism got up to full speed after a prior half-empty page, the servo paper drive logic with speed detection and feedback loop would go into oscillation (at about 200kHz, from memory), superimposing a massive high-frequency signal onto the D.C. drive voltage to the paper-drive motor. Unfortunately, the wiring from the control logic to the motor was held in the same wiring loom that carried data from the data socket at the back of the printer to the printer buffer memory. The printer had a 1-line buffer that would be accepting the next line of print from the computer as it was printing the current line, and so Bingo! .... When the servo paper drive system went into oscillation, the radiation into the data logic wires scrambled the data for the next line, which was of course the first line on the next page. (Webmaster's note: Yes I know, I know, why don't I just let Rob tell his story? Well he introduces so many pertinent observations of a whole field of work that no longer exists, due to the vast majority of machines these days being throw away replaceable items. Additionally the not so technical readers, will not be aware of the significance of writing which presumes a technical readership. Having justified my continuous interruptions, for the not so technical readers, this is an interesting fault. An obscure circumstance burst of electromagnetic radiation, inducing interference in the neighbouring data wires, due to a hunting closed loop servo system as a result of mechanical slack, caused by time related wear, is not the first thing that comes to mind when considering causes of such a fault.
When the printer was new (and tested by DEC), there was no slack in the motor drive belt or bearings etc., and the logic kept tight control over the speed of the paper movement as it followed a smooth ramp up in speed until it (maybe) hit full speed, then a smooth ramp down in speed, leaving the paper at just the right spot to start the next page's printing. Beautiful when it was new. But as the printer aged and some mechanical slack came into the system, if the paper drive got to full speed, the feedback would try to keep it 'idling' along at the same speed, which set up the scene for oscillations when the control was designed to be too 'tight'.
I determined that as the paper movement had to follow only a smooth ramp up and down in speed, over a period of 1 to 2 seconds, the feedback loop (which worked well) didn't need to have a frequency response of 200kHz. So I simply lifted a wire connection in the feedback signal, inserted a modest value resistor (probably 100 ohms, for example) in series, then put a small capacitor (say 0.1 microF) after the resistor, to earth - the simplest low-pass filter you can have. Bingo again! ... The printer still did its Form Feeds reliably, but no more oscillations under any circumstances, and hence no more scrambling of printed data. (Webmaster's Note: I just can't help myself, it just strikes such an accord with me! I love Rob's solution. There is an acronym, that I have heard in multiple fields of engineering KISS: "Keep it simple stupid". This is a simple but effective solution and for that reason I think it is the best. )
I tidied up the modification, wrote up the problem, its cause, and the field change I had done, and sent it all to DEC - whom I hope took action to modify their design and retrofit any change to existing printers. I had to put the same change into the second Harold Park printer and the 2 for Wentworth Park.
It was fortunate that the problem showed itself before the systems went live, as our only record of the betting details, totals and dividends that the system produced were the printouts after each race, on the 2 on-course system line printers. Remember - no discs in those days.
David Hamilton, who was the ATL NSW operations manager at the time the PDP8 system was introduced also remembers the introduction of the PDP8 system in his tote memoirs in the Memories of an Ops Manager and Harold Park chapter.
Harold Park Paceway has been host to many great trotters including Cardigan Bay, who won a million dollars in Australasia and the United States, and Lucky Creed who won 23 successive events. Their performances will be talked about at Harold Park for as long as people remember the time Dan Patch broke the two minute barrier for a mile at Yonkers. And that will be forever.
Harold Park Paceway was the first racecourse in the Southern Hemisphere to install an Electronic Totalisator System.
ATL did the job, its tenth successful Electronic installation since Aqueduct. This year there will be 89 race meetings at Harold Park, 47 of them for trotters and 42 for greyhounds. All will attract huge crowds, for this racecourse is the most important venue of its kind in Australia.
The N.S.W. Trotting Club, the company controlling Harold Park, at the moment can boast of having the most modern Totalisator facilities anywhere in the trotting world.
I have included the second extract from the company document ATL International Name in Totalisator Betting Systems, as Rob Stone mentions being the installation engineer for Wentworth Park as well as Harold Park.
Twin computers combine on-course and off-course investments at Wentworth Park. Closed circuit television is used to transmit Odds, Results and Dividends information throughout the Course for both public and internal use.
As an added feature, the running of each race is videotaped and played back to the patrons on monitors 2 minutes after the greyhounds have crossed the finish line.
The Computer Room incorporates the TAB off-course and ATL control offices in a fully air-conditioned area which is located in the Paddock Totalisator House and overlooks the track.
The Electronic Totalisator System is capable of serving 128 Ticket Issuing Machines selling Win, Place and Quinella tickets in the range of 50 cents to $50.
High speed Line Printers record the information stored in the computers. This information is available for checking and for record purposes at any time.
Logging Teleprinters record all instructions given to the computers and type out data as specified by the program. Instructions not taken care of by the program are keyed into the Totalisator Control Console which serves as an input device.
Calculations of Odds and Dividends are made by the computer under program control. The Lamp Boxes in the Indicator Board are activated by the Board Control Unit which is part of the computer complex.
Five selling divisions equipped with type J11 single value Win, Place and Quinella Ticket Issuing Machines serve the betting needs of the public. Both types of machine are fast lightweight press button units designed to efficiently and speedily handle business at the selling windows.
A new grandstand with many special design features is envisaged in future plans for Wentworth Park. ATL shares with the Wentworth Park Trust and the National Coursing Association their enthusiastic anticipation of this plan becoming a reality, thus providing another facet to the modern facilities offered at this internationally-known greyhound race track.
I attended the DEC training course with Allen Lakeman in Crows Nest, with Bill Johnston and Ray Stone. (Webmaster's note: Ray is Rob Stone's brother.)
(Additionally I have included the following reference to Wentworth Park as this system followed shortly after the Harold Park system and is mentioned by Rob Stone above.)
I recall the problem Rob Stone had with one of the high speed printers at Wentworth park. He spent many hours analysing the fault that turned out to be the printer drum which consisted of two print wheels on a single spindle one of the print wheels had loosened and moved out of alignment thus printing the wrong characters.
George Jenkins was also involved with me on the Wentworth Park CCTV system. We had completed the CCTV installation and it was working satisfactorily, then one morning we discovered a 50 cps hum bar rising horizontally on every monitor. I was at a loss to know where the interference was coming from, I called George to come and assist. It turned out that over the weekend the electrical contractor MW Ellison had altered the power points in each monitor housing adding earth wire from the GPO to the actual housing. The City Council Inspector had issued a E notice requiring the monitor housing be earthed, unfortunately the old Ledger grandstands electrical wiring was leaking to earth this ground voltage was the cause of the hum bar. The added earth wire was totally unnecessary as the monitor housings were bolted to the steel grandstand structure. A quick trip around the housings with a step ladder and a pair of side cutters did the trick.
I don't think MW Ellison's electrician at WP would question the Electrical authority's direction to earth the monitors GPO to its steel housing even given the housing was bolted to the steel grandstand structure. I knew the MWE electrician very well he was a Dutchman ex POW in Germany during WW2, he knew more about the Tote cabling of Randwick, HP & WP tracks than ATL did. He could terminate large W/P cables onto those ATL open frame terminal frames as neatly as the Women in the factory. Frank was a excellent artist, he had a love of aircraft especially fighters, he could do a wonderful water colour painting of a dog fight with English and German planes so realistic. Frank lost his life, when a power pole at Randwick collapsed.
At that time the two inspectors that regularly did inspections of works carried out by the on-course electricians, were a Mr Dear and a Mr Love. (Truly!) They even had leather bound copies of SAA3000 the bible of electrical work standards. I had studied it back in the 1950's when at Tech. ATL did follow its directives when it came to power connected manufactures. Anyway one day Messrs Dear & Love turned up at Harold Park. We had just finished and were testing the new large Ledger indicator. They came into the indicator and looked around at the systems and the 100 or so lamp panels. They sighted the data cables terminal block coming from the computer room on the paddock side of the track, "What are those Green wires?" they asked, They are part of a set of five data connections that fed the indicator with its 2 of 5 code. "You cant have green wires as green is always a earth wire" they said and began to write up a E sheet. The reason the were insistent on us changing the green wires was that the Paddock computer room was on a separate Power Grid system to the Ledger. About this moment, Ron Hood turned up and somehow convinced the inspectors that it was lunch time and he invited them to dine with him at the local Harold Park Hotel. I never did see an E sheet, or for that matter, did I ever come across Messrs Dear and Love again!
Webmaster's Note: Ron Hood was my first Manager when I joined Automatic Totalisators. I thoroughly enjoyed working for him as he afforded me the time to do a lot of study relating to ATL products and their engineering. So long as I had my head in a manual, an engineering drawing, or a diagnostic listing, he was happy. I learnt a lot from Ron!