Nice article in Forbes Magazine: “Wall Street’s Speed War” about the latest project I have been working on.
On September 1, I will be representing Spread Networks in a webinar on“Ultra-Low-Latency Networks & Financial Trading – Fast Networks for Fast Markets” with Ciena and Tabb Group.
I just completed a Q&A with Ciena, posted at: http://blog.cienacommunity.com/n/blogs/blog.aspx?nav=main&webtag=cienablog01&entry=78
I have been working with Spread Networks, which has executed on a very simple but exciting idea to reduce trading latency: Find the shortest possible path between the major exchange hubs in NY and Chicago, dig a trench, lay conduit in it, and pull a new state of the art fiber optic cable.
For more detail, take a look at spreadnetworks.com
In recent posts I have commented on topics in electronic trading on the one hand, and telecommunications on the other. It is time to unite those two threads.
I have also been following discussions on LinkedIn regarding low latency networking, and have been working on related projects with clients.
Building a low latency network is not a “black art”, but it does require a solid understanding of how networks are built, as well as an appreciation for the tradeoffs that telecommunications carriers make when building networks.
I will start with some general principles and then focus in more detail on the specifics.
The first thing to understand is that any engineering project involves trading off objectives. As the old joke goes, “I can build it fast, cheap, or reliable, pick any two.”
Let’s look at some examples for the design of networks:
Most telecom networks (i.e. networks built by AT&T, Verizon, etc.) are designed to scale well, to VERY large numbers of users. After all, network construction is capital intensive and tends to reward firms that can service a large number of users. Additionally, networks generally follow Metcalf’s law, which says that the network becomes more valuable as the number of users increases.
But the design choices that enable scale are bad for latency. A basic design principle to achieve scale is to employ hierarchical designs. Rather than attempting to connect every user directly to every other user (in a “full mesh”), hierarchical networks collect traffic locally, aggregate it, deliver it to regional concentration centers, and then route traffic back out to local centers and out to the users. This kind of “hub and spoke” design (looking much like a family tree) scales well, but forces traffic to take paths that are longer than necessary and to pass through routers and switches that add latency. If you have spent any time transiting ORD, DFW, or ATL, you will know what I mean. What works well for airline economics (and helps keep fares low) isn’t so great if you are in a hurry.
More generally, telecommunications carriers optimize their networks for:
- Transporting large amounts of bandwidth. This is no surprise, as most telecommunications revenue is ultimately denominated in units of bandwidth.
- Multi-service provisioning, i.e. the ability to carry (and sell) many different services on one network. Carriers make substantial investments in their networks, and they want to spread those costs over as many customers and services as possible. T1s, T3s, cell phone backhaul, traditional phone calls, text messages, internet traffic… The more services, the more revenue. To achieve this, carriers build networks using layers of routers, switches, muxes, and multi-service access nodes. These enable carriers to provision everything from T1 to text messaging, but add overhead and latency.
- Servicing population and business centers. Willie Sutton robbed banks because that is where the money was. Telecommunications companies build networks to serve population centers. When laying fiber from NY to Chicago, telecommunications companies try to pass as many population centers along the way. So a typical network that carries traffic from NY to Chicago might go up to Bighamton, over to Buffalo, then down to Cleveland before it goes on to Chicago (or down to Philadelphia, over to Pittsburgh, up through Akron and Cleveland, etc..) Take a look at the cable routing in any country, and the lines are far from straight; they zig and zag to stop by as many cities as possible. Who cares if it adds a few milliseconds, your text message will still get there fast enough.
- Cost. The cheapest way to build long-haul networks is to use existing right of way, which for the most part means rail lines. Qwest started as a railroad spinout, so did Sprint. Conveniently, railroads also zig and zag to visit key population and economic centers too. Try to take an Amtrak from NY to Chicago without passing through Buffalo, Pittsburgh, or Washington DC (not to mention Albany or Philadelphia); You can’t do it, and neither do most telecommunications companies.
- Traffic patterns that are “typical”, e.g. peaks from “American Idol” messaging or Mothers Day. Since most of the carrier’s money comes from a very broad market, they do not design their networks to consider the latency requirements or traffic patterns (e.g. peaks at US Market Open rather than Mothers Day) of trading.
- Operational efficiencies… For example, telecom companies routinely “re-groom” circuits to free up capacity on certain links, to facilitate maintenance, etc.. There operational convenience might mean a sudden change to your latency!
None of this is intended as a criticism of the traditional carriers. It is a simple matter of the engineering and economic tradeoffs that drive the large carriers to build networks optimized for the people who pay them the most money. And electronic trading is a very niche market for them.
Networks that have been built specifically for financial markets (e.g. BT Radianz, Savvis) do somewhat better. The engineers who built those networks made tradeoffs differently than telecom companies usually do, optimizing for trading traffic, focusing on the geographic markets that matter to financial services, and focusing on services (e.g. IP multicast) that facilitate market data. But while these networks do better than general purpose telecom networks, they are still not ideal for the current generation of high frequency, low latency trading. While they were built for trading, they are still trying to address a larger marketplace than the ultra low latency HFT market, and they are international in scope. For example, one of the main objectives of RadianzNet was to provide the ability to rapidly connect financial customers to one another, and so a large “community” was key to the success of that objective. But scalability to large communities gets in the way of reducing latency.
So when BT Radianz developed its low-latency network “Ultra Access”, and when NYSE developed SFTI the engineers removed layers of routing and generally made decisions that favored reduced latency at the cost of scalability. (For comparison, BT Radianz’ RadianzNet was designed to scale globally to tens or hundreds of thousands of clients, where BT Radianz Ultra was designed only to scale to thousands of clients in localized geographies.)
So how do you build a low-latency network for trading?
Well the first conclusion is that if you REALLY care about latency, you are going to have to build it yourself. While networks like Radianz and Savvis are great for quickly connecting counterparties and they reduce the IT load relative to firms that manage their own networks, if you really want the lowest in latency a shared network is just not going to cut it.
So where do you start? First, simplify the topology. While hierarchical designs are cost effective and enable scale, the lowest latency design is a simple point-to-point network. In practice for most firms that will mean multiple point-to-point links, one to each venue. That is more expensive than traditional networks, and doesn’t scale well, but there are only a handful of low latency matching engines to consider.
Next, strip out every possible layer of equipment from the design that you can. The fastest solution is a direct connection from a LAN port on a server (e.g. 1G or 10G Ethernet) directly to the WAN link.
What kind of a WAN link? In almost every case this will be either a “lit” wavelength provided by a telecom company, or a dark fiber that you light yourself (a traditional T3, OC3, OC12, or similar service will add overhead that you don’t want.) By buying a lit wavelength (1Gig Ethernet or 10G Ethernet wavelength), you are eliminating almost all the carrier overhead (SONET Muxes, Multi-Service Access Nodes, etc..)
You may even want to go further and buy dark fiber. Simply put, dark fiber is fiber installed by a carrier that you light yourself. It may seem that there is little difference in having a carrier light the fiber versus lighting it yourself, but there may be a number of benefits to leasing dark fiber and lighting it. All things being equal, dark fiber will provide a lower latency solution:
- Dark fiber allow you to isolate traffic from different trading strategies and different matching engines onto separate wavelengths at minimal marginal cost (i.e. once fiber and equipment is leased/purchased, cost of adding wavelengths is typically very low.) This eliminates all possible sources of queuing delay that can occur when multiple trading strategies are using the same wavelength.
- You can select your own equipment to light the fiber, and there ARE real differences in latency among different equipment vendors and configurations. (Carriers rarely choose or deploy optical equipment based on latency, focusing much more on the number of wavelengths that can be supported, variety of interfaces supported, operational considerations, etc..) If you light it yourself you can optimize for latency rather than optimizing for bandwidth and multi-protocol support.
Plus you know you have control and security, since you have dedicated fiber that nobody can tap, and you can have confidence that your network will not be re-routed or re-groomed.
Of course all things are not always equal. At this level of engineering, speed of light becomes very important. (The speed of light in fiber is not the number we all learned in high school, i.e. 299,792,458 meters per second. Depending on the refractive index of the fiber, the speed of light in fiber is roughly 2/3 the speed of light in a vacuum.) So a “dark fiber” route from NY to Chicago that is 100 miles longer than an otherwise comparable “lit” wavelength could be 1-2msec slower (round trip) because of the increased mileage. But all things being equal you should be able to get lower latency and less queuing if you light it yourself.
So when designing your wide area links, consider dark fiber, but be very careful to understand the routing of that fiber, the optical mileage (which is longer than the route mileage to allow for some slack in the fiber) and the type of fiber (which determines things like the refractive index). With lit services, make sure to get actual latency measurements, not just SLA numbers (that are usually padded to minimize risk for the carriers.)
Building an ultra low latency network isn’t a black art. It just requires an in-depth understanding of how networks are built, right down to the fiber routing, and it isn’t the least expensive way to go.
Larry Tabb made a comment the other day about my post on Google’s strategy that got me thinking… Larry said “It used to be that we were supposed to care about privacy – seems more and more that we are willing to give up privacy for functionality, ease of use, and the ability to better understand what I as a consumer want.”
The same is true about trading.
When deciding how much personal information to reveal on Google Buzz or Facebook, the correct answer will depend partly on who you are (there is a big generational element to this), how much information you have to protect, and what you want to achieve.
If you are 18, have very little private information, and want to socialize as much with as many people as possible, then you are likely to reveal a lot (maybe too much) on Facebook.
If you are an adult professional (with a lot more at stake) and you just want to know what your former college buddies are up to, you might reveal a lot less.
So with trading.
Every decision about trading tactics (i.e. how to get the trade done, not whether to buy or sell) should be driven by the investment strategy. For example:
- If you are buying a small quantity of a highly liquid stock because you anticipate positive earnings, then you might well choose to cross the spread and pay the offering price to get your order filled quickly before (you hope) the stock spikes up. Because your order is immaterial to daily trading volume, you have little to lose and (you hope) a lot to gain by showing your entire order to the market.
- On the other hand, if you are a large fund trying to accumulate a substantial portion of a small-cap (and probably illiquid) stock which you intend to hold for 2-3 years, you are more likely to let your order sit in a dark pool, go to a broker for capital commitment, slice your order up using an algorithm, etc.. In general you will be more passive and less likely to show your hand. You certainly wouldn’t show the entire order on the exchange, because you have a lot to lose and little incentive to rush.
The European regulation that addresses “best execution” is MiFID. MiFID recognizes that different investors have different strategies and therefore will make different trade offs in their trading tactics… They do not have a single/uniform definition of “best execution”, any more than Facebook has a single definition of “appropriate privacy.” Instead, MiFID recognizes that best execution is not limited to execution price but also includes cost, speed, likelihood of execution, likelihood of settlement and any other factors deemed relevant.
In contrast, the US regulation that addresses “best execution” (RegNMS) defines it very narrowly as getting an order executed at or between the best bid and the best offer on a publicly displayed market. Never mind that you may want to do 100,000 shares of a small cap name and there may only be 100 shares at the inside quote.
In practice, traders in the US markets need to make trade offs and answer a variety of questions, including:
- How important is certainty of execution
- How important is speed of execution
- What is an acceptable market impact
- How much of their hand to show
One of the most important tradeoffs is between how much of your hand to show (resulting in market impact) versus certainty and speed of execution. i.e. If you show your entire order to the market you have a much higher probability of getting it filled quickly, but you will move the market.
It is instructive to consider some of the recent innovations in the US equities markets (or innovations that, while not strictly recent, have become more popular in the last few years) in light of these trade offs.
Take algorithms. Algorithms allow a trader to slice a large order into many small orders and spread them out over time, usually to be executed on displayed markets that only offer liquidity in small quantities. The trader can thus comply with best execution obligations (i.e. each individual “child” order is executed at or inside the NBBO) while minimizing market impact and information leakage. (If you were to look at the average price achieved by all child orders that make up a single parent order, the parent order may well be executed at an average price better or worse than the NBBO at the time the parent order was created. So if the client were buying and the market rose dramatically over the life of the parent order, did the client get “best execution” just because each individual child order looks good?)
Dark pools provide another alternative. In general they offer less certainty or speed of execution, but also lower market impact. An order may sit for days or weeks in a dark pool until it is matched. (Your mileage may vary… On a pure institutional/block system like LiquidNet, the order is likely to sit until it is matched by a contra-side institutional order. On Millennium, it may be executed incrementally as it is matched against algorithms, but exposure to such smaller orders also creates greater risk of information leakage and gaming.)
Now let’s take a look at flash orders.
In a nutshell, flash orders allow a client to send an order to an exchange, and optionally permit the exchange to briefly “flash” the order to liquidity providers who can fill the order at a better price. This improves the certainty of execution, and may provide price improvement over the NBBO, but it increases information leakage. Should the client care? Well, if he is not concerned with information leakage (maybe he only had that one order) and he wants to “get done” now, then he is probably happy to use a flash order. If on the other hand, his order is one of many child orders for a larger parent order, he may prefer to be more passive and decrease information leakage.
Some teenagers expose all their photos and personal information to the world (including, ahem, college admissions officers), while others only permit their friends access to their information.
On the face of it, this is good. Flash orders provide a useful tool that allows a knowledgeable investor to make a reasonable tradeoff.
But consider Google Buzz. Google Buzz was rolled out last week as Google’s social network, its answer to Facebook and LinkedIn and MySpace. Cleverly, Google leveraged users Gmail contacts to automatically create “followers” on Google Buzz, and (by default) allowed any user of Buzz to see any other users followers. Google very quickly found out that many users don’t want the world to know who all of their Gmail contacts are. Especially if that list includes doctors, lovers, bookies, or other relationships best kept private.
So how does this relate to Flash Orders?
Google’s Buzz appears to be a reasonably clever service, and it actually does allow users to control their privacy. But Google erred by making the default privacy settings very non-private, and by making the privacy controls too difficult to find. They made a decision on behalf of their customers to expose those customers information. They made the trade off.
Likewise, the problem with flash orders is really not flash orders themselves. The problem is that in many cases a broker chooses to submit an order as a flash order, with the client never realizing (never mind understanding) the trade off that the broker has made on the clients behalf. Of course this is just a symptom of some more fundamental problem, namely:
- RegNMS dictates a very narrow definition of best execution, which has had the unintended consequence of encouraging innovation that helps the customer achieve their execution objectives despite this regulatory constraint. Maybe we should have a broader definition of best execution that incorporates the clients trading objectives.
- Brokers incentives are not aligned with their client’s objectives. Which is why brokers receive rebates for order flow, use flash orders, and generally do things that maximize profits for brokers rather than for customers (the phrase that comes to mind is “where are all the customers yachts?”)
- Even asset managers, who act as the proxy for most investors, have incentives (maximize assets under management) that are at cross-purpose to their client’s objectives (maximize risk adjusted return.)
But those issues are subject for another post.
Recently I have been focusing heavily on trading and market microstructure, but my other hat is in the internet/telecommunications world, and yesterday’s announcement by Google piqued my interest.
Why would Google announce that it is getting into the broadband business (see http://www.google.com/appserve/fiberrfi/public/overview)?
To understand that, you need to understand what is strategic to Google (for background, Chris Dixon sets the table very nicely at http://cdixon.org/2009/12/30/whats-strategic-for-google/)
Microsoft wants you to live in Windows and Office, hence anything that makes those less relevant that (see e.g. Netscape) is a threat.
Apple wants you to live in MacOS and iPhoneOS, buy your media through iTunes, etc… (Apple is really a lot like Microsoft in that way, they just execute a lot better… and a lot of that has to do with the fact that they never thought of themselves as just a software company like Microsoft, they think of themselves as a total integrated experience company.) Anything that disrupts or taks you away from that total integrated experience (as insanely greatly designed by Apple) is a threat (see e.g. flash). This is why Apple bought Quattro Wireless (a mobile ad company): so you can consume ads within apps on your iPhone/iPad instead of on the web… or more to the point, so that iPhone/iPad app developers have a way to monetize their apps and thus an incentive to develop for iPhone/iPad instead of for the web.
So what does Google want? Google wants you to live on the web (unlike Apple or Microsoft which really want the web to be secondary to their platform) where they can deliver targeted search-based ads. Let’s consider some strategic moves by Google:
- Google and Apple are inherently at odds with each other, since the more time you spend in (for example) iPhone OS and iPhone apps, the less time you spend in the browser (looking at Google delivered ads.) Which is why Google is promoting Android, a web-centric phone OS. When you use an iPhone app, you aren’t on the web consuming Google ads (the internet is there in most apps, it is just buried as a communications layer that you don’t see.)
- Google wants fast, ubiquitous, cheap, and open internet connectivity, which facilitates you spending time in web-centric applications (like Google search, Docs, Google mail, Google Apps…) and viewing ads delivered by Google. This is why Google pushes net neutrality… That commoditizes internet access and prevents ISPs from disintermediating Google. That is also why Google bid for wireless spectrum… to push the market to provide open access over wireless.
- With Google voice, the web is the platform… Since I started using Google voice, I find myself spending much more time in Google contacts, placing calls via the web, etc… Google voice has shifted my phone experience to the web. Where it sells ads…
So why is Google “planning to build, and test ultra-high speed broadband networks” (which, by the way, will also be “open, non-discriminatory, and transparent”, i.e. an embodiment of net neutrality.)
Not because Google wants to be a telco.
Partly because Google wants to create some competition to the incumbent cable companies and telecom companies. But Google can’t create a material level of competition.
What Google can do is to create pressure, through the media and through regulators, on the cable companies and telecom companies. That pushes those companies to provide high-speed, open broadband networks (which, not coincidentally, make it much easier to live on the web, happily using Gmail, Google Apps, and consuming Google ads.) Imagine your congressman or city councilwoman asking the local cable company when they come up to renew their license “Why can Google offer an open internet service with 1 gigabit fiber-to-the-home connections when you only offer a crummy slow 5 megabit connection as part of a bundle with 6 movie channels????” Watch the cableco executives squirm as they try to explain that one.
So does Google want to be a telecom company? No, they want to offer proof points and create pressure for faster and more open internet connections, so you can live on the web and consume ads delivered by Google.
Not that there is anything wrong with that.
On February 4, I will be speaking on a panel in Chicago on best practices in high frequency trading. Great location, since HFT is often associated with cash equities, but a lot of the interesting opportunity (and a lot of the action) is in futures and in arbitrage between futures and cash markets.
More detail at: http://worldrg.com/showConference.cfm?confcode=FW10009
On December 9 I will be chairing a panel on “Emerging Technologies Enabling High Frequency Trading”. Still lining up the panelists, with some great candidates and looking forward to an interesting panel.
On November 18, I will be chairing a conference on Dark Pools and chairing one of the panels.