Unlike Star Wars, both the light side (lit wavelengths) and the dark side (dark fiber) can be good for your latency. There is no single right answer, so in this article I will explain them both and highlight their respective strengths and weaknesses. Let’s start with the basics.
Dark Fiber:
In the beginning there was darkness. All fiber starts out dark, i.e. strands of fiber optic cable with no signal. Telecommunications companies acquire rights of way (roads, rail lines, etc.), install conduit, and install fiber optic cable(s) in the conduit. Cables may have from 100-800 strands per cable, each of which is “dark” when first installed. Since spools of fiber have limited length, on all but the shortest routes strands are spliced together to create a continuous path for light to travel from end to end. Cables also have “slack” along the route, to accommodate expansion and contraction of the earth, bridges, etc. (like expansion joints on bridges) and to make it easier to splice cables back together in the event of a cable cut. (If there were no slack in the route, cable would be much more likely to stretch and break under tension, and it would be more difficult to splice the cable back together when it gets cut.)
Lit Waves:
To actually make all that fiber do something useful, telecommunications companies take pairs of strands (one for each direction that signals will travel) and “light” them with optical transmission equipment, literally sending waves of light along the path. Data is encoded and transmitted over these waves, and may be presented by the optical transmission equipment in many standard formats such as Gigabit Ethernet, 10Gig Ethernet, and SONET (i.e. OC-3/12/48/192). They then sell these as Ethernet or SONET services, or use them as transport for other services (e.g. IP/MPLS networks, voice transport, etc.) Typically the optical transmission equipment used today is DWDM (Dense Wave Division Multiplexing) equipment, which allows telecommunications companies to transmit many (potentially hundreds) of waves on a single fiber pair, using different “colors” or frequencies of light for each wave. (Since DWDM so dramatically expands the effective capacity of fiber optic cables, it has made it much more viable for telecommunications companies to sell unused strands of dark fiber.)
For routes longer than about 50-60 miles, things start getting a bit more complex. As light travels along fiber optic cable, signals deteriorate, more so as there are more splices or if the quality of those splices is poor. So telecommunications companies construct small data centers along their routes to house equipment for optical amplification (which takes place entirely within the optical realm) and regeneration (where the signal is converted to an electrical signal and a shiny clean new optical signal is generated.) Just as highways have on-ramps and off-ramps to pick up and drop off traffic along the way, and interchanges to other highways, so too do long haul fiber networks. In the case of fiber networks, telecommunications companies locate switching centers where it is convenient to interconnect with the telecommunications equivalent of on-ramps and off ramps (cell towers, internet peering points, local phone and cable companies, etc.) and to interconnect with other long-haul routes, using ROADMs (reconfigurable optical add-drop multiplexers) to provide the optical switching.
Do you choose the dark side or the light side?
So what does this mean for businesses that care about low latency? As you might expect, there are some tradeoffs.
Dark Fiber: In general you will get better latency, more control, and greater economic scalability with a dark fiber network, at a cost of some complexity and initial expense.
- With a dark fiber network, you get to make some of the decisions that the telecommunications companies would make on your behalf if you were buying lit waves. You get to select the optical equipment and how it is configured. You get to decide whether and where to use optical amplification vs. regeneration, and how many waves to configure on the network. It is your private network and you control it. As you might imagine, most telecommunications companies will opt for design decisions that maximize the amount of bandwidth and the number of services they can sell (since this maximizes their revenue), while you may choose to optimize for reduced latency.
- You can also eliminate devices altogether (such as ROADMs) that add latency but don’t add functionality that you need.
- While you can’t control it, you have greater visibility of the physical routing of your network, the type of fiber used, the amount of slack, and the location of facilities used for amplification and regeneration.
- The initial cost of a dark fiber network is likely to be higher than the cost of a single lit wave (less so on short routes where equipment is a greater percentage of the overall cost than the fiber), but the cost is largely fixed. If you own the fiber, the incremental cost of adding waves is very small, making it much easier and cheaper to add waves (e.g. when adding a new trading strategy, data feed, or trading desk.) For large institutions, the total cost could well be less than buying a large number of individual waves.
- The complexity of operating a dark fiber network is greater than that of buying lit waves, although some dark fiber providers will either operate the network for you or have partners that provide outsourced operations.
- Since fiber cables are point to point and there aren’t routes between every possible pair of end-points, you may need to integrate multiple routes from multiple dark fiber providers to connect all of the end points you need. If you are focusing on low latency trading, the dark fiber providers typically have routes that interconnect the key exchange colocation facilities, reducing the need for you to do this integration.
- Finally you probably want resilience in the event of fiber cuts or equipment failures. With dark fiber you have the benefit of knowing your exact fiber route, so you can ensure true diversity, but you also have the responsibility of sourcing and implementing an alternate route.
Lit Waves: With lit wave services most of the complexity is hidden from you and the initial cost is lower, but the latency will generally be higher, the costs will not scale as well, and you lose some control.
- With lit wave services, you are one of many customers using a fiber pair that the owner wants to manage for maximum revenue. Since most telecommunications markets are not particularly latency sensitive, most telecommunications companies optimize their network design to maximize bandwidth, the number and type of services, and the number of interconnection points (each of which represent a revenue opportunity). All of these add latency.
- Since you are not consuming an entire fiber pair, your initial cost for a wave will be much lower, and if you only need a small number of waves your total cost will likely be lower. But as you add waves, your costs can grow rapidly.
- A lit wave service is engineered, operated, and managed entirely by the telecommunications company. Your provider has already deployed and managed the equipment needed to provide the service, and integrated the various fiber routes necessary. This simplifies your life, although it reduces your visibility of the network. For example, the operator may choose to “groom” the circuit (move your traffic from one fiber to another, possibly to an entirely different route.) This can affect your latency and could compromise your diversity. With large telecommunications companies, even your account team probably doesn’t have visibility of the actual routes used to deliver your service.
- Telecommunications companies can provide lit wave services that integrate multiple underlying fiber paths. As a result, they can deliver to more end-points than most dark fiber providers, and they can provide “protected” services (composed of two or more diverse routes).
Both dark fiber and lit wave services will provide you with better latency than other alternatives such as IP or MPLS networks, and there is no single right choice for every application or every business. Understanding the differences is a crucial first step to making the right decision for your business.
Proton Advisors 