How far away from Six Sigma is the global automated trading infrastructure?

A Six Sigma process is one in which 99.99966% of the products manufactured are statistically expected to be free of defects (3.4 defects per million).  Six Sigma is considered to be world-class quality.  So, the question is:  what would it take to get the automated trading industry to be at the Six Sigma level?

To properly answer this question, we need to decide how to calculate a software error rate. This is a very interesting question since it is based on standard quality-metric calculations: divide the number of errors by the total number of messages to the bid-ask queue.

To calculate this number, we should take the total number of errors divided by the total number of messages.

Let’s assume that the total number of messages is approximately 300,000,000 messages per day per large exchange.  For simplicity sake, let’s also assume there are 5 large exchanges in the world.  So, the total number of messages per year might be approximately 378,000,000,000.

From a cursory scan of news items, it appears that from 8 / 1 / 2011 to 7 / 31 / 2012 there were 31 “possible” problems worldwide.

If we assume that a single error contains 10,000 bad messages, then the total number of errors is about 300,000 messages per year.

The calculation of the software error rate is then 300,000 / 378,000,000,000 = 8.20E-07.

The calculation for a Six Sigma process is 3.4 / 1,000,000 = 3.40E-06.

Therefore, based on the simple assumptions, the quality of the automated trading industry exceeds the Six Sigma level.

My question is then, where did I go wrong in my assumptions?

The second question is, could it be that the problem is not the poor quality of trading industry, but rather the sensationalizing of the errors by the media?

What does the map with blue and red dots show?

MIT researchers worked out the physical locations to minimize network latency between a trading system (the small blue dots) and principal exchanges (the big red dots).  The graphic is also a good visualization of the extent and integration of global financial markets.

Wissner-Gross and Freer,”Relativistic Statistical Arbitrage,” Physical Review E 82, 056104, 2010.  Figure 2.


What does a quality management system in AT look like?

For automated trading, a quality management system includes processes to achieve prudent research, design, development, operation, and control of AT systems. This covers critical activities including quantitative modeling, risk control techniques, backtesting, simulated trading, and probationary trading. This also includes processes for and documentation of software and hardware testing that prove the firm has demonstrated that an AT system functions properly, is operationally safe, and robust to behave acceptably during potential extreme events. Statistical methods for evaluating the stability of AT systems and for real-time monitoring have been developed.

What’s the process the firm should go through to do all these things to justify their belief in the stability of the system? The K|V methodology (i.e. a study of methods) is such a methodology (see Quality Money Management, Elsevier/Academic Press, 2008).

AT 9000 is agnostic with respect to research, development, operation, and control methods. Thus, as a study of methods, K|V is not a prescriptive method itself.  Nevertheless, all firms engaged in AT do (or should) engage in the activities described, though not necessarily in sequence of stages and steps shown.  Firms should perform in their own study of methods, and define internal processes that satisfy a quality policy and quality objectives. These processes will be unique to each firm and its organizational environment, and potentially to each AT system R&D project. The intent is not to imply uniformity in the structure of an AT firm’s quality management systems or uniformity of documentation.

The ability of AT firms to prove the stability of their systems also depends upon the availability of execution venue (exchange) simulation facilities to fully test those systems. Such simulation facilities must enable testing against all manner of extreme market and infrastructure events.

By achieving a QMS that follows AT 9000, automated trading firms should be able to satisfy their organizational obligations to prove and document that its AT strategies and technologies will operate safely and profitably. There is also a wide body of literature demonstrating that the use of quality management systems improves financial performance.

What is a quality management system?

A quality management system (QMS) for an organization is a defined process and related artifacts necessary to implement a quality policy. Quality management is the orchestration and control of operations so that they satisfy customer expectations and obligations to stakeholders. The ISO 9000 family of standards is the most widely recognized quality management standard. Many industries where societal safety must be ensured use ISO 9000 as baseline to consistently and efficiently meet customer requirements, regulation, and broad social responsibilities. For example, here are somne industry-specific QMS that use  ISO 9000 as a baseline.

  • Aerospace: AS 9100
  • Automotive Suppliers: ISO/TS 16949 and VDA 6.1
  • Chemicals: Dow Chemical Company Quality Management System Manual
  • Environmental: ISO 14000 family
  • Medical devices: ISO 13485 and IEC 62304
  • Food safety: ISO 22000
  • Health care: the US National Committee for Quality Assurance (NCQA)
  • Telecommunication Systems: TL 9000
  • Testing and Calibration Labs: ISO 17025



What is the AT firm’s organizational responsibility?

The SEC and the CFTC have recently lowered the bar for proving market manipulation from intent to recklessness, implying (in the case of AT, necessarily organizational) imprudence or irresponsibility.  So, in the case of failure of an AT system, how can the organization prove it was responsible, that it was prudent in its AT research and development (R&D) and operation and control (O&C)?  The answer is they were responsible because they followed a recognizably prudent process, one that proved and documented that the firm was justified in believing the future performance (i.e. stability) of its AT system.

AT systems make decisions based on proven research.  As such these systems can only modify the outcomes of these decisions using the structures embedded in the software (i.e. real-time risk control).

How do you know your trading system will work?  What passes as proven research?  The obligation of the AT firm is to prove and document that an AT system’s trading strategy and technology will operate in line with expectations and to specification.  Prudence demands that the firm prove that its systems will run in control.  This obligation can be satisfied by following a prudent process that justifies expectations in the performance of the AT system.

What are examples of conflicts in AT development and operation?

The need for low latency gives rise to a conflict between speed (necessary for profitability) and the inclusion of fail-safe code that may add latency (necessary for safety of external stakeholders).  An inherent conflict also exists between minimizing costs and satisfying obligations to, for example, paying for research and development of real-time risk controls and/or redundant systems.  As time to market for an AT system matters, production pressure also lead to launch of risky trading systems.  The need for profitable AT systems cannot take precedence over the quality—stability and reliability—of the global system.

What risks do AT systems pose?

Each AT system is a piece, a proprietary technological component, of the global trading network. The performance of such components affects potentially all markets, either directly or indirectly.  An out-of-control AT system can flood a market or markets with order requests on a time-scale that precludes human intervention.  Such flooding can affect market prices, the profitability of other trading firms and exchanges, as well as societal confidence in the sustainability and safety of the financial markets.  The strategic or technological failure of an AT system could be catastrophic for these stakeholders in the global trading network.

Is automated trading ethical?

Financial markets enable price discovery. Price information and price discovery are generally considered to be good for the public.  As most AT systems make use of limit orders, they provide liquidity to financial markets.  Limit orders add information to the market.  AT systems add to price information and, therefore, price discovery. Several academic studies show that AT increases liquidity and decreases volatility.  Thus, this activity cannot be construed as inherently unethical.

Rather than the ethics of AT, AT 9000 consider ethics in AT.  By ethics we mean those standards of conduct that apply to AT firms, and the primary ethical component is responsibility.

What is High Frequency Trading?

Here’s a definition recently discussed by the CFTC. High Frequency Trading (HFT) is a form of automated trading  that employs:

a) algorithms for decision-making, order initiation, generation,  routing or execution;

b) low latency technology that is designed to minimize response  times, including proximity or co-location services;

c) high speed connections to markets or order entry;

d) recurring high message rates (orders, quotes or  cancellations), using one or more forms of objective measurement,  including cancel-to-fill ratios, participant-to-market message  ratios, participant-to-market trade volume ratios.