Dan Noland

The previous blog article about server deployment mentioned our two physical (stratum one server) deployment projects: Project Ikenga and Project Tonatiuh.

Much in the same way we dove into virtual servers, we figured it would be fairly easy to find an underserved country and find a company who would lease us some power, space, and network bandwidth.

Oh, how adorably naive we were… :)

After much trial and error, investigation, and a couple wake-up calls from some...let’s go with “well-traveled”...friends, we gradually came to realize that there is a lot more to nailing down a location than finding the nearest person happy to take our money.

Multiple times during our search into locations, we’d find a deal-breaking detail. We then would take this new information and check to see if any of the other locations had that issue as well. During this evolution of understanding, we developed a “checklist” of sorts that contains several different dimensions of what’s involved with a deployment. It’s unlikely to find a “perfect” location, but the goal is find the locations with the most acceptable tradeoffs.

The following is our current, subject-to-change, guide for locating the best cities to lay down physical gear. As we have been adding things to consider to the list since the search started, it’s likely that it will continue to change/expand as time goes on.

1. IS-95A cell phone network: it turns out that in order to participate in a cellular network, the cell towers have to have hyper accurate time.. While you do need a subscription plan to send messages or make phone calls on an IS-95A network, you do not need one to read the timestamps on messages sent by the cell towers. Several stratum one NTP server vendors take advantage of the fact that just being within the broadcast range of a cell tower means you can have incredibly accurate (nanosecond-level) time accuracy.

While we also have GPS-based servers on hand, we feel going for a simpler, cheaper deployment for our first one makes a lot of sense. IS-95A CDMA-based gear is affordable and transportable enough that the company has a couple sitting on a shelf in my closet.

2. Datacenter Provider We Can Have Confidence In: This was a fun lesson. We were swapping emails with a potential provider in Africa (a location that we discovered would fail more than one of the tests on this list -- thankfully BEFORE we arrived!). The company had a nice website, were very responsive to our emailed questions, and seemed very eager to become our hosting partner. As we did more research on logistics, we thought to look at the location using satellite imagery, only to discover it was a literal hut with a satellite dish. Needless to say, we realized that vetting both the personalities AND the location/facility were required for future exchanges.

3. Health risk: A lot of underserved countries present risk for diseases that we in the United States are lucky enough to put out of our minds Thankfully, the Center for Disease Control gives a list of recommended vaccinations for every country on the planet. It was a bit of a shock to realize that for some of the locations we were going to require vaccinations for things like typhoid, yellow fever, meningitis, hepatitis A/B, cholera, and/or malaria.

Some of these vaccines require scheduling your appointment weeks or months in advance, and some take weeks or months after receiving the vaccine for the body to reach maximum immunity. All of this needs to be factored into travel planning.

4. Safety risk: we leave it to the State Department’s travel advisory information to help inform this dimension. Obviously we have a strong preference for level 1 (“normal level of awareness“) but are willing to consider countries up to level 2 (“increased level of caution“). As this is a passion, not a job, we don’t see a compelling reason to disagree with the U.S. government when they encourage people to “reconsider travel” or “do not travel” to an area.

5. Visas: PublicNTP has a preference for countries with no requirement for business visas to enter their country. It’s not a big deal, but it does add complications and costs.

6. Accessibility: we’re not looking for an Indiana Jones-grade adventure to get to our server box. For now we’re looking for locations with a major international airport with good connectivity (if indirect) from where our Directors live: Utah, Indiana, and the Washington, DC Metro Area.

7. Costs (Travel): Costs of airfare, hotel, and food need to be reasonable. When committing to a deployment, we’re wanting to remain on-site for anywhere up to a couple weeks to make sure things are up and running. More people + more time = more costs we have to anticipate.

8. Costs (Datacenter): we’re not big enough or influential enough (yet?) to build our own sites and maintain them. So it’s in our best interest to work with people and businesses who already have a datacenter. Their one-time and recurring costs for installing and hosting our servers need to fit within the budget our donors make possible.

9. Connectivity: we want to deploy our time servers where they will reach the widest audience possible. A big part of that is what the network connectivity from the datacenter looks like. It’s very common, for example, to see boomerang routing between locations very close to each other in South America. We’ve seen a number of internet routes from, for example, Bogota, Colombia to São Paulo, Brazil that route through Miami, Florida in the United States. It makes no logical sense that you’d have to go north--the exact opposite direction--before being able to go back south. These sort of network access issues affect the suitability of hosting platforms for PublicNTP.

10. Import/Export Restrictions: due to US and international law, it may not be legal for us to deploy equipment in a given part of the world.

As an illustration, here is the export list for Microsoft products. Note that the XBox One has an Export Control Classification Number (ECCN) of 5A992.c, assigned by the US Department of Commerce. As it has been classified, there are certain countries where one would have to apply for a license before taking an XBox One to those countries (in this case, the only countries it would not be legal to export the hardware to is on the Department of Commerce’s Sanctioned Destinations list).

On the flip side, there are countries where it may be illegal to import the equipment. One reason for such restrictions might be that the equipment contains cryptographic capabilities.

PublicNTP has retained legal counsel to help us ensure that we abide by all import/export laws of the US and any countries where we may deploy systems.

This list is subject to change because we’re always learning more. Still, these basic guidelines have already helped us vet various locations, make quick decisions, stay safe, and triage/prioritize some of our first possible location.