Definitions: High Frequency Trading, Flash Orders, Dark Pools, Algorithms

As I have been following the commentary in the popular press on high frequency trading, dark pools, etc., I have noticed a lot of confusion on the terminology and what these things mean.  In particular I have seen reporters, public officials, and others talk about program trading when they really mean algorithmic trading, criticize high frequency trading for the (perceived) sins of “flash orders”, and generally conflate all of these things together.  A recent article in the NY Times ( for example, says “Powerful algorithms — “algos,” in industry parlance — execute millions of orders a second and scan dozens of public and private marketplaces simultaneously. They can spot trends before other investors can blink, changing orders and strategies within milliseconds.” Actually that isn’t what “algos” are, at least not in common industry parlance.

In general the confusion stems from the fact that computers play a role in all of these, and so they tend to all get lumped together.  In order to have a healthy debate on US equity market structure, we should all have a good understanding of what these things are, so I will use this post to explain some key terms.  Others may use these terms differently, but I have outlined below what I consider to be widely accepted (by actual practitioners) definitions.

One important distinction to understand with any computer based trading is the phase of the trading process that is being automated.  At a high level:

  1. Pre-Trade: An investor (whether an individual investor, a portfolio manager at a fund company, or a computer program acting on behalf of an investor) performs some analysis that leads to a decision on whether to buy or sell a stock.  This is where “alpha” is created.
  2. Trade Execution: Once that decision has been made, a trader (or a computer) is responsible for implementing that decision, and uses discretion to decide when to place an order to buy or sell, what type of order to place, where to place the order, and the size of the order.  It is important to understand at this phase of the cycle that the trader is not deciding whether to buy or sell, but how to do so in the way that best meets the investment objectives that led to the buy/sell decision.  (For example, if the investor decided to buy because he is hoping for a positive earnings report the next day, the trader may want to buy more aggressively to get into the market ahead of that news.  If the investor decided to buy a large block because he has a 5 year view on the companies outlook, the trader may take a slower and more passive approach to avoid bidding up the price of the stock.)  The ideal execution tactic should support the investment strategy.
  3. Matching: Once the trader places his order, a variety of different mechanisms may be employed to actually match a buy order with a sell order.  Before the widespread adoption of computers, this matching function was performed by specialists and floor traders on the floor of physical exchanges, and by market makers and brokers over the telephone.

For each of the different techniques described below, I will identify where they fit in this simplified three-step process.  Since these different techniques evolved over time, I will describe them in (roughly) the historical sequence in which they appeared in the market.

Program Trading: Whether you call it Program Trading, Basket Trading, or List Trading, it is one of the oldest forms of trading using computer technology.  Often used as a term by the media to describe ALL forms of electronic trading, “program trading” best describes when a trader submits a list (or “basket”) of orders for simultaneous (or near simultaneous) execution.  Program trades can be used to achieve a number of investment objectives including transitions (when a plan sponsor moves assets from one money manager to another), rebalancing, moving funds into or out of an index, etc.. “Program trading” is fundamentally a mechanism to execute a series of trades across a portfolio of stocks.   The New York Stock Exchange defines program trading as “a wide range of portfolio trading strategies involving the purchase or sale of 15 or more stocks having a total market value of $1 million or more”.  While program trading is generally automated today through the use of computers, the fundamental strategies (e.g. of tracking an index) preceded widespread automation.  Automation just makes program trading faster and easier.  More important, program trading is used as a mechanism to implement an investment strategy.  It is not a strategy in itself, and therefore fits into phase #2 in the process outlined above.  Portfolio Insurance, commonly blamed for the market crash in 1987, is one (but only one) application of program trading.

Electronic Trading: While “programs” could be executed with minimal computer technology, and were around even when most trades were executed by specialists on the (physical) floor (or market makers in the over the counter markets), the next phase in automation was when computers were introduced into the actual process of matching buy and sell orders.  So the first proper use of the term “electronic trading”, and still its best definition, is the use of computers to  match orders, i.e. step #3 in the process above.  In the US, services such as Globex and Instinet were pioneers in electronic trading.  Exchanges in Europe were among the first to go electronic, eventually followed by the US exchanges (in part by merging with existing non-exchange based electronic markets.)  Some of these systems were and are continuous real-time systems, others are “point in time” matching systems (or “crossing networks”), and while they take different approaches, they are all fundamentally matching buy and sell orders electronically.  Today most trading of simple securities such as equities is electronic, with lower rates of technology adoption in markets where instruments are less well standardized (e.g. credit derivatives.)

Dark Pools: While the name sounds sinister, Dark Pools developed to automate the function of the block trading desks that used to mint large amounts of money for sell-side firms.  Originally called “Crossing networks” (early examples included Lattice, ITG Posit, the Instinet Cross, etc.) they matched (or “crossed”) large blocks of stock.  Today they include ITG Posit, Pipeline, Goldman Sachs Sigma-X, UBS PIN, Credit Suisse Crossfinder, NYFIX Millennium, and many more.  The appeal of dark pools is not that they allow traders to hide surreptitious or illegal activity, but that they allow traders to buy and sell large blocks of stock without moving the market.  Originally dark pools only matched large blocks of stock.  More recently dark pools opened up to algorithms (see below), which allows traders to expose large blocks of stock to the orders generated by algorithms and without moving the market (because the block of stock is not displayed, nobody will see a 10,000 share sell order suddenly on the NBBO and watch the price drop in reaction.)  They allow algorithms the opportunity to trade against the large blocks and benefit from price improvement (i.e. the ability to get a price in-between the best bid/offer).  Some dark pools are operated by independents (e.g. LiquidNet, Pipeline), others are vehicles for “internalization” by brokers (i.e. they allow brokers to trade their customers orders against each other and against their own inventory, providing opportunity for price improvement and reducing exchange fees.)  Dark pools implement the third step (matching) in the 3-step process described above.

Algorithmic Trading: Once the function of matching orders was automated, networks established to connect to these markets, and programmable interfaces such as FIX were developed (as opposed to dedicated screens which is how early electronic platforms were accessed), it became possible to automate the function of delivering orders to an electronic market.  One of the jobs of a trader is to manage the flow of orders into the market so as to achieve “best execution” (a topic for another post).  With electronic interfaces in place, it became a (relatively) straightforward process for programmers to develop automated systems that took an order from a customer or portfolio manager, sliced that order into smaller pieces (which would have less market impact) and send them into an execution venue.  This function is performed by computer algorithms, and came to be known as algorithmic trading.  Those purists with degrees in Computer Science may protest that ALL functions performed by computers are executed by algorithms, and an introductory course in algorithms is mandated in all Computer Science curriculums offered today.  Mathematicians would argue for an even broader application of the term algorithm, and they would be right.  But in the trading world, the term “algorithm” is generally understood to mean automation of the (very tactical) process of placing a (usually largish) order into the market, often by means of breaking it into smaller chunks and managing the timing of those “child orders” into the marketplace so as to achieve a particular objective.  (That objective is generally formulated as a benchmark to be tracked, such as Volume Weighted Average Price.)  In other words, algorithms are used to automate step #2 in the process above, and contrary to the assertion in the NY Times:

  • They don’t “execute millions of orders a second” (they generally spread the execution of an order out over hours, or even days and weeks, placing “child orders” into the markets with intervals of minutes or hours.)
  • Just like human traders, they do “scan dozens of public and private marketplaces simultaneously” both to assess the amount of liquidity in the market (e.g. to avoid placing orders too large or too frequently and thus cause prices to move) and to determine where best to place the order.
  • They do try to detect “trends before other investors can blink”, but primarily to avoid getting poor executions, e.g. to avoid accidentally buying at the peak of a transient spike in price (not to scalp investors).
  • They generally don’t “change orders and strategies within milliseconds”, and while they might change orders (e.g. if the market starts moving against them) they do so to implement a clearly defined strategy (e.g. “buy 100,000 shares passively without moving the market” or “sell 10,000 shares at the daily volume weighted average price” or “buy 20,000 shares at as close a price to now as you can.”)

Algorithmic trading came into place for four reasons:

  • First, algorithms are simply a way of automating what traders already did.  That is to say, looking at multiple markets and determining where best to place an order (called “smart order routing” when computers do it), and breaking large orders into smaller chunks that can be released into the market at the optimal time.
  • Second, as the trading process has come under the microscope to be measured, and as benchmarks such as VWAP have been widely adopted, algorithms provide a simple and low cost way to execute against a benchmark.
  • Third, as the buy side has assumed more responsibility for its own trading, algorithms provide a low-cost way to execute trades across multiple brokers without hiring large trading staffs.
  • Finally, RegNMS has imposed “best execution” rules that require trades to be executed at the National Best Bid/Offer, which has resulted in smaller orders being shown at the NBBO, and as a consequence has driven traders to slice institutional orders up into retail sized chunks (to match the orders at the best bid/offer.)

Strategy or Black Box Trading: While algorithms are focused on the tactics of trading (i.e. given a decision to buy or sell a quantity of a security, how is that decision best effected), strategies or black box trading systems are one step higher in the food chain.  Such systems continually scan streams of market data, analyze them for patterns, and make decisions on whether and how much of a security (or usually a set of securities) to buy and/or sell.  They fit into step #1 of our 3-step process.  This includes strategies such as “High Frequency Trading” and statistical arbitrage (which may or may not be high frequency).  They are quantitatively driven techniques, implemented using high-speed computers.

High Frequency Trading: High frequency trading, very simply, encompasses a range of trading strategies (and therefore fits into step #1, i.e. pre-trade) that involved the rapid buying and selling of securities (and often the rapid posting and cancellation of orders as well.)  Broadly there are three classes of strategies pursued.  These strategies are not exclusively high-frequency, although they are used by high frequency traders (among others):

  1. Automated market making, where the HFT trader posts buy and sell orders simultaneously, makes some money (maybe) on the spread, and makes some money on rebates paid by exchanges in return for posting orders.  Like the market makers of old, HFT firms make money on some trades, lose on others, but expect to make a net profit across a large number of trades.
  2. Predictive traders, where the HFT employs software that does try to “spot trends before other investors can blink” and like all momentum traders, try to buy before the price has run up and sell out before it crashes back down.  In a way they are like the many day traders in 1999 who bought internet stocks in the expectation that prices would run up, and tried to sell them before everyone headed for the exits.  While this time the game is measured in milliseconds, the winners are still those who bet right, and get out early enough.  Of course some are stuck holding the bag after the price has collapsed.  And this time it is all done using computers.
  3. Arbitrage traders, who look for short-lived inefficiencies in the markets, buy the (relatively) undervalued asset while simultaneously selling the (relatively) overvalued asset, and unwind the trade when prices come back into an equilibrium position.  Simple examples are pairs trades (i.e. a simple pair of securities where some price relationship should hold such as options with differing durations, two different share classes of the same stock, etc).  More complex examples are statistical arbitrage, where there is a relationship between complex baskets of securities.

Unlike algorithmic trading, where computerized techniques are used to establish or exit from a long or short position, and where portfolio turnover ranges from high to low (with average holdings potentially multi-year), high frequency strategies are neither long nor short but market neutral, portfolio turnover is extremely high short (average holding periods measured in seconds or milliseconds), and the strategies aim to end the day “flat”.

Direct/Sponsored/Naked Access: As trading has become more electronic, many buy-side institutions have chosen to take on the trading function that was historically performed by sell-side traders.  In some cases this is to better control the execution of their trades, in other cases it is simply to reduce costs (or for the broker to reduce costs by pushing clients from full-service to a low-touch model.)

  • The first, and still most pervasive form of this is direct market access, where the broker provides the buy-side institution with some combination of a terminal (execution management system) and connectivity, the client sends orders electronically to the broker, where they pass through the brokers order management, risk management, and compliance systems, and on to the exchange for execution.
  • As clients became more sensitive to speed and latency, many brokers offered their clients sponsored access, where the client connected directly to the exchange, using the brokers membership and clearing through the broker, but bypassing networks that route from the client through the brokers data center and on into the exchange.  Instead the clients either collocated with the exchange or connected directly.  In this model, the client still sends their orders through the brokers risk management and compliance systems, either through software written by the broker (e.g. Lime brokerage, which also provides the hosting/collocation) or in many cases by specialized vendors such as FTEN.  This software is deployed at the same location as the buy-side institutions servers (typically collocated at an exchange).
  • Finally, there are some clients who have an intense need for speed, and who have built systems that are as lean and fast as possible.  These institutions use “Naked Access”, where they collocate their servers at an exchange, connect directly into the exchange using the brokers sponsorship and clearing, and where there is no brokerage system/software in-between performing risk management or compliance functions.

Flash Orders: Probably the most provocative innovation in 2009 (although the concept is not new, and has existed in derivatives markets well before being adopted by the cash equities markets) is the flash order.  The idea of a flash order is very simple:  A trader can (optionally) send his order to an exchange or ECN and specify that the order is a flash order.  When the exchange receives the order, if it cannot be immediately matched, it is “flashed” (shown) electronically for a very brief period of time to firms that have signed up to receive flash orders.  Those firms have a brief amount of time in which to respond to that order with a matching order, which allows the original order to get a better price than it might otherwise have received.  If none reply in time, the order is routed out to another exchange.  The firms that receive flash orders are typically High Frequency Trading firms pursuing an “automated market making” strategy (see above).  It is important to note that the firms sending flash orders are typically not high-frequency traders, and only some high frequency traders choose to receive and respond to flash orders.  Flash orders are an interesting example of where one firm (the firm submitting the flash order) is focused on the tactics of trade execution (#2 in the 3 step process) and seeking a good quality execution, and is interacting with another firm that is trading in the market as an inherent component of their strategy (i.e. the HFT firm).

Hopefully this brief explanation provides a framework for understanding and assessing the various technology innovations that are stirring debate on our public markets.

(Disclosure:  I sit on the board of Marketcetera, which offers products to the strategy trading marketplace, and has relationships with the NYSE, Lime Brokerage, and others.)

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: