Bringing the Internet to new places around the world is important, but it's also very challenging and perhaps even a daunting task as well.
So with us today is a very special guest, Darwin Costa, who has spent years and years doing just that, working with service providers, submarine cables, technical leaders around the world to bring higher speed, higher quality Internet connectivity to new places where there really wasn't anything significant prior.
So in today's episode, we're gonna dig into Darwin's experience, hear some success stories, some challenges that he faced, and also learn a little bit about how Darwin went from being a pro soccer player to the prolific network engineer he is today.
My name is Philip Gervasi, and this is Telemetry Now.
Darwin and Doug, it is great to have you both on the podcast today. Darwin, getting to see you for the first actually, I think we met once before over Zoom. So so for the second time. And then, Doug, of course, it's always a a pleasure to speak to you. Now I understand that you and Darwin actually have a long, relationship, and you've been friends for quite a while. So why don't you take it from here, Doug?
Sure. Thanks, Phil.
So as mentioned, I'm Doug Madory, the director of Internet analysis at Kentik, and, and we've got Darwin Costa presently at DE-CIX, an old friend of mine, and we've been talking about, Internet and submarine cables for for a number of years.
Darwin, would you introduce yourself and talk about, like, what's your where you at now and your your role at DE-CIX?
Yeah. Sure. And first of all, thanks guys for for the invitation. I really appreciate that. You know? Good to be here on the podcast with you guys.
So, yeah, like Doug said, you know, we know each other from, from previous lives, I would say. My previous life is previous lives. So I was working for Motorola Cables for seven years, and we were actually deploying subsea cable, the first and unique subsea cable between Angola, Africa, and South America. And then eventually in two thousand nineteen, I joined DE-CIX. And I do look over activities in, Africa and, Latin America as an interconnection lead person.
So I wanna tell we'll have to tell the Saks, the South Atlantic cable system is the is the cable, Darwin's talking about and tell that story. But before we get into that, Darwin, can we, let's, let's talk about your, you know, where did you come from? Where did you, how did you get into computers? What what, or this this industry? Like, bring us back to beginning. So you you are you grew up in Angola. Is that correct?
Yeah. Right. I think it's always good to start with that. You know, that's an often question which is is not asked, actually, you know, and, let's go into it because I do face a lot of problems when people ask me where are you from, you know, because eventually, they will think I'm a Brazilian, I'm from Egypt, or from Morocco.
But, no, I was born and raised in Angola, And then, yeah, in the capital city of Rwanda. And then eventually, Angola was a country with thirty years of civil war, you know, and my, young ages of, elementary school, you know, I always like to to hang out at school, you know, doing, some homework and hang out with friends and play a lot of soccer. You know? So my mother used to actually to beat me a lot because I was more into soccer than anything else.
Then eventually, you know, the war broke out into the capital city, and, I was young. And, luckily, I could play some football, and I told my mother, hey. Why don't you send me somewhere? You know? Maybe I can I can do something for the entire family, you know, in the future? And then, eventually, I went to Spain, to Gran Canaria. That was the first country I ever went when I was a young child and playing football.
How how old are you at this?
Yeah. I was, yeah, I was something like eleven, twelve years old, and I was playing for the was already playing for the national team of Angola, hundred and fifteen year years, old age.
And we went to a a football tournament, which is a tournament, which is called, the World Cup, you know, for young children. And then eventually, we went to Gran Canaria, and, I did play it well, you know, and then one of the scouts was there, and he told my grandfather because my grandfather was with me. And he told me, you know, he told, can you leave your grandson over here with us, you know, so to play some soccer and to to develop and to to grow further? And Then eventually, I told I told myself, you know, I said, shit.
I need to buy a I need to buy a gift for my mother so she allows me to stay in Spain. Eventually, eventually, I did that. And when I got back home, she said, you know what? You are going nowhere.
You you you are made for school, so you need to go to school. And then, you know, my dream, my grandfather's dream was, like, falling apart.
But as far as the war was getting worse, you know, she really thought, you know, I need to send him somewhere.
And then I had someone, an uncle of mine, who was living in Netherlands back then. And I always watched this kind of movies of the football schools in the Netherlands, and I said, this is kind of cool. Why not? Right?
So I did went to the Netherlands, you know, and, I did some kind of, you know, these, tests that they have in the summer, in the winter. Unfortunately, I came into the into the wintertime, and it was freezing cold, you know, and I couldn't I couldn't do anything. You know? I was just not used to it.
And so my uncle said, you know, let's wait for the summer. It's more yours your your kind of weather. You know? And then eventually, I went to a first division club, and, you know, things went well, and they said, you know, you can stay in here.
But on the other side of the ocean, you know, on the other part of the continent, my mother was always beating me up over the phone. You need to go to school. Don't drop school. School is important, you know?
And, you know, it's it's totally a different world when we talk about Internet and then professional sports.
But one of one of the things that I learned a lot, and I'm very thankful that I did both at the end of the day, was was really disciplined, you know, because discipline is very, very important. You know? And although I wanted to become a football player, I knew I need to do something with my studies. And then, eventually, I liked a lot video games.
I think all of us did, you know, in some center point in our lives. You know? Sega Mega Drive, and whatever. You know?
And so I said, hey. This is fun. You know? So why don't I do some something with computers?
And, you know, back then, computers were just just coming up, you know, and all of these I think one of the first one of the first curses I did back then was the CCNA, the Cisco one. You know? It all it was all new to me, but that's fine. You know?
Open the PCs and having a new motherboard and doing stuff with the PCs, a new memory, and blah blah blah. You know? And so, eventually, I long story short, in two thousand nine, I was selected to to train with the national senior national team in in Angola.
I went down to Angola at my trainings, and then I was selected to play for the Africa Cup, which used, was played back then in two thousand ten in Angola.
Unfortunately enough, I, I got malaria when I got back home. And so and so I, I was two weeks, you know, in the two weeks in the hospital, and I actually had two choices. You know? Just finish school, which was about five months to to get it done, university, or I, would try to recovery, you know, and then, eventually go back to football again.
But, you know, I just followed followed whatever my mother said. You know, school is important. That's what I did. You know?
And So, so and and in the Dutch system, you may I feel like maybe you told me once you you made it to the second, you you made the the second division, pro, team.
Which team was that?
Yeah. It's a it's a team called, well so I will do it in English and then in Dutch because in Dutch, you said, but in in in English, you said, a a g o v v. That's what you call it.
Okay.
It was a team where I played with guys like I will give you a name.
He's still a good friend of mine, but, Driss Martens is playing for the Belgium national team.
You know? So, yeah, we all we all grew up over here, you know, because the first division team, what I used to go when I was young, was called Vitesse, in another city. But, eventually, when you get a little bit older and, they don't have any kind of responsibility with you anymore when it comes to school. Right?
So I wanted to do both. And in order to do that, I needed to go to training and go to schools during the night or whatever. You know? I needed to do my own scheduling, and that's what I said.
You know? I think discipline is was was really key to me. You know? And I didn't want any have anybody fell down, especially my mother.
You know? And, you know, eventually you know? You make a lot of friends in sports industry as you might as you guys might know because in US, the sports industry is huge. Right?
If you watch to NBA game, even the MLS right now, it's it's kind of crazy. Right?
So when you have these kind of dreams, you know, and we don't when you don't go to school and things actually don't unfold as you would like, then things become a little bit worse. Right? So I do have friends which are still playing, but I do have friends which are, you know, just working in a factory. Nothing bad against that, but it just it just it just life. Right?
I'd say that, you know, at the, conferences at Darwin's at Darwin's a popular guy. He's a very cool dude, and a lot of people wanna hang out with him. And I think he's also too humble to talk about the fact that he was a professional soccer player, and there might be a, a telemetry now first.
Phil, correct me of I don't know any of the professional athletes we've had, on the on the the podcast.
But I can't think of even one, so I I agree.
It's a really neat thing. It's a neat, you know, and and, you know, especially for such a great guy. But alright. So we we can move off the soccer thing. And then, so how did you how did you wind up at Angola Cables?
Yeah. So that's that's another good question, and thanks for raising that. You know? So when I was done with my school and, eventually, football was nothing was nothing more for me. You know? I met my wife during holidays in Portugal.
And then she told me, listen. I'm living in Angola. And I was like, I'm not moving away from from the Netherlands and going back home. No.
I'm not doing that. You know? I I have my life over there. But when I met her, I totally decided to go back back to my own country because, you know, I was done with studies in in the Netherlands, and then, eventually, my mother heard something about submarine cables, Internet, and then she told me, hey.
Maybe there is something you you you could look into it. Right? And I was like, okay. Cool.
Then I tried to join to do, you know, the the best of both worlds, you know, getting well, fell in love with my wife and then eventually going back home to be with my parents, which, which we all left them. You know, there is one particular case in Angola, and I think many in Africa countries, the same tank, which is, you know, when you're when you're going to high school, you know, then if your parents have, you know, a little bit of good conditions to send you away, they will do it, either to Portugal, either to whatever. Right? For us, Portugal was the entry point because we were colonized by by Portugal, so it was the obvious, country to go when when everybody wants to to do something with the with high school and university up.
And my my my way was a little bit different. I went to the Netherlands and, you know, it was totally different than than the common path that everybody takes in Angola. And so, yeah, when I went back to Angola, you know, I was trying to apply for work, and with my IT background, you know, the work that I just received was just, you know, help desk and all of these things, you know, in my young career. You know?
That's actually what you need to do. You know? You need you need to have your hands dirty, you know, and doing doing this kind of ticketing jobs and stuff like that. But, eventually, you know, I wanted something else.
And when Angola Cables came into the picture, they were looking into, well, younger generation, which could speak many languages, you know, because it would be a multinational project and so on and so forth. And then I applied, and, eventually, they they hired me.
How how long had the how long had the company, Angola Cables, been in existence at that point? Was it is it relatively new?
No. The company was created on paper, if I'm not mistaken, two thousand ten, something like that. And then, eventually, two thousand eleven and twelve, things started happening.
Okay.
Does the does the government so the Angola is a a a country that has a an oil oil wealth as a source of income for the government.
Right.
Is there government investment in things like, Angola cables and, etcetera?
Yeah. Yeah. No. I mean, you know, one of the as you mentioned, you know, aside of oil, there is there is a lot of another natural resources, you know, which come into play, but, obviously, oil is the main source of income for the for the entire country. And when Angola Cables was created, it was actually created by the five major shareholders, which are a joint of public and private, shareholders, but mainly were were were public. Right? So the the one Yeah.
That was my understanding that some of the some of that oil wealth was translated into, investment into the generally in the IT telecommunications infrastructure and then specifically to Angola cables, to underwrite this investment to Yeah.
You're right.
Build something for the country.
Okay. And so then how so you're you're starting off at, help desk, IT one zero one in, Angola Cables. At some point, you're involved in the, become involved in the building the first, major submarine cable across the South Atlantic, the South Atlantic cable system.
How did you get involved in that project?
Yeah. I have to be honest. You know, I, I think IT is a very broader. No.
No. Our our industry is very broad. Right? I I I do like to use the the example as a doctor.
Right? If you go to the hospital, you have a lot of doctors which can perform different activities of operations or whatsoever. Right? And, eventually, when you go back and you started somewhere in Europe, you go back home.
You know, your first question you got from your mother, the printer is is damaged. Can you make it? You know? So in the head, it's like, okay.
So you started the IT, so it will fix everything. And so, eventually, when we started to turn our cables, we started working in a in a house, actually, because we didn't have any kind of offices. And so I was actually trying to do whatever I took to help the company. Right?
Because I was number eleven within the company. And so there was me and someone else, another guy, which had some IT skills, and we were setting everything up, the most servers, and so on and so forth. But, eventually, there was also a lack of, product folks, business folks, and so on and so forth. And my first call, I have to be honest, you know, my first call I had with the international company international company because we needed we needed to buy IP transit from from a tier one or tier two network back in the days.
You know, when when I was in the call, you know, and the guys were talking during the call, I went back home and I told my mother, I don't know which kind of secret services those are, but I never heard anything about this, you know, because the subsea industry to me was pretty new. You know? And I kind of kind of see that back when I for instance, I went last week to my daughter's school to give a presentation about subsea cables and blah blah blah. And they were like, does this exist?
You know? So it's it's a surprise for everybody. So that was my first feeling when I when I when I when I joined the company and on the first meetings.
So what so what year roughly would that be?
What was that?
What year, are we talking about?
What year? Two thousand eleven. I think two thousand eleven.
I think I think also, we have to remind I mean, you know this very well, but just for anybody listening, like, the so the continent of Africa connects to the global Internet through submarine cables that run along the east and west, coasts of the the continent. But, all of that is relatively new. Like, there like, the first, submarine cable that went down the, East Coast was, was, like, fifteen years ago, and prior to that, it was entirely satellite. And then there's, like, there's a little bit more history on the West Coast, but that too is not that long ago. Like, we were all, you know, adults when, the continent of Africa was entirely connected through satellite, which is, you know, constrained capacity, high cost, high latency.
And and it was really a new day when these submarine cables came into, came into being and the and the and the satellite industry kinda collapsed, around, you know, each each time a landing station came alive. You know, we were watching this in routing back in two thousand nine when a submarine cable landing showed up. Like, in East Africa, you could just watch the, everybody move off of satellite and go over to and and it never came back. It wasn't even there as a dormant backup. It was just completely gone, just trashed.
But, so, I mean, the fact that people would be be like, oh, what are these things? I mean, we, nowadays, people are still don't know what they are, but but they were actually we're really, you know, only a few years old, depending where you were in in Africa.
So True.
True. True. No. And just to just to add into that as well, Doug. You know?
But and whenever I do, I like to do a lot of, you know, not only professional talks, but also family and friend talks. You know? I really like to explain them what is all this all about. Right? The Internet that has been built for years now, you know, and all these protocols that we run on the Aussie model whatsoever, you know, young generation, nowadays, they don't want to know anything about that. The only thing they want to know is, you know, it's really is it working?
You know? And if not if not, then if you try to explain them, they will easily tell you, listen. I'm playing online. I just need to get back online again.
You know? And then if eventually, for us, the old schools, you know, kind of stresses me a little bit personally, right, because I really want to to to try to explain you why is it not working and where where is the server located or whatever the issue is. Right? But, eventually, it is what it is.
I think this, generation see, what what we call them or what they call themselves, you know, it's very into get it get get it down now. Right? They don't they don't kind of care where is it coming from, where is it going to. But, yeah, going back to your point about Africa, you know, it's it's relatively new.
And if you're still looking into it, you know, every time I have the chance to do a talk in Angola, and I will do another one end of the year, you know, where the prime minister will be there as well. You know, I always try to to do some comparations. Obviously, they don't like to see comparations like, you know, a European country or US country with with Africa, with Angola, for instance. Right?
They will say just just use the same you know, another country in in in Africa. And then and then eventually, I try to do that. Like and I the the example that I still use is, like, look into how many cables are landing landing in South Africa and in Nigeria.
Right? And look how many cables Angola has for a thirty million population. Right? I mean, you know, if you look into Portugal only, there is more than fifteen cables active by now. So Portugal has ten million people. Do you really think they're building this all of this capacity for ten million people, whereby sixty percent of these ten million people are elders?
No way. Right? They they they they're just trying to make sure that the digital transformation strategy that the government or that the country has really move forward as it should be. Because, I mean, you know, what we are doing currently in Angola, you know, we are adding one cable while the other is already dying. And I give you an example. Two Africa is coming up, and SAD three will be dead by, I don't know, many months or years from now, but it's pretty old now. Right?
So SAD three.
So Yeah. I'm not sure when that came into that's the old cable that has the least amount of, capacity. And, yeah. So I think, like, when I was first doing this, you know, fifteen years ago, when we see a cable cut, around Africa, it just meant the country was completely offline.
And I think maybe, it was the Red Sea cable cuts in February twenty twelve, that we saw. Like, in East Africa, cable countries would I mean, there was a lot of connectivity that was lost, but there was actually, at that point, multiple cables, and there there was some some element of resilience where providers could stay online through switching over to another cable, which was kind of kind of a new thing. It was still very dependent on, severing cable. So it seemed like there was, like, a trend going in the right direction of increasing Internet resilience.
But if I I I'd be really interested in your your take on, you know, how far or not far we've gotten because I think if we look at the cable cuts that took place in March, off the coast of Cote d'Ivoire, the so there was a there was an incident.
Our as our understanding is, there was some sort of, like, undersea landslide or some kind of thing that happened on the seafloor, and it took out Mhmm. Four four cables. There's a lot of cables and really messed up a lot of connectivity up and down the East Coast. Sorry.
The West Coast of West Coast.
Yeah.
Yeah. West Coast of Africa.
And, and what was interesting to me is, you know, in in all this time, obviously, we've built there's been a lot of submarine cables we've built that's improved, kinda international connectivity resilience. There's also a big movement with, you know, content and local hosting and stuff and things moving, into, the countries.
But it's it's interesting to hear when when when those cables went down, like in Nigeria, there were payment systems that wouldn't work anymore and all kinds of Right.
It's like local local services were no longer working.
And, it does make you wonder, like, so what are those dependencies that we haven't been able to move into the country? It seems like we still have despite despite all the the cables, and despite all the local, you know, coasting local caching of content, we still have a lot of, you know, these the countries still have a maintain a lot of dependencies on things that are accessed through the submarine cables.
I don't know. We is that, is that just how is it gonna be or is there, I don't know, if that's a, you know, if we're gonna get to a point where, you know, a cable could go down and and most services are still accessible except for those it seems like there ought to be things. Like, I I think I think it was, like, you know, there's, like, some authoritative DNS, you know, things. You you find all these you discover all these dependencies you didn't know you had once you lost the connection, and it turned out, you know, like, the, there's, like, a payment service that the authority of DNS was back in Europe, and now it's unreachable.
And now everybody's things even though this is a service that really is just used in Nigeria, it doesn't operate. And, it's, like, some of these things are solvable, but we're still we still have this, I don't know. Maybe that's maybe that's just something we we're still we're still working through. I don't know.
Do you have any opinions on that that arc, that evolution, as you've seen it?
Yeah. I think I think, you know, you you very briefly touched it in into multiple points, and I will get back into the DNS stuff in a few minutes ago. But, you know, obviously, when those kind of things happen, obviously, that, you know, multiple factors will then be analyzed as you guys normally do at Kentik as well. You know? And everybody tries to to do kind of analysis to understand what it what it went wrong and what can we do to, you know, to to avoid future future impacts, such as these ones. Going back to March two thousand twenty four, you know, I think that was the time when Equiano came to rescue. Isn't it?
It is. So Equiano is a Google cable, that survived, and it was also because it was relatively new, it had probably the the largest amount of capacity.
Well, there's a couple things that came came came to the rescue. So Equiano was probably number one.
Number two is the Sachs cable.
So this is the cable, you know, that Darwin was involved in building a cable directly across the South Atlantic from Angola to Brazil. We still gotta talk about that story.
But that that gives them the option then to instead of you know, the primary path for traffic is to go up and down, north south up to, like, Portugal, Western Europe, but then you've got this, you know, direct shot over to Brazil where you can kind of, like, ricochet off Fortaleza up to the, West Coast or the East Coast of the United States. I got my East West mixed up today.
And and so I could see from a routing perspective that that shift of, going to, you know, through from losing losing the connectivity from one of those cables that was cut and then going out to the, to Brazil.
And then even the SAIL, cable, which was a competitor, to Saks, the Chinese backed was it South Atlantic?
What does it stand for? Internet.
I know.
I I'm not quite sure what it's saying.
What sales stands for. Okay. But that goes to Cameroon. So that goes from Fortaleis to Cameroon.
Right. And, and that too, we could see, traffic shifting. So so I think Equiano probably gets the number one credit. Saks maybe number two.
Sale, also took took on, we can see cam Camtel in Cameroon switching to Right. Going through Cogent, which I didn't realize they were buying Cogent in Fortaleza. But you could see it from trace routes. You could see that that you're going across this.
So, but, And whatever and whatever the credits are, dog, you know, it was it's good to to emphasize here to to our listeners as well.
You know? I I actually exactly what you described, you know, was exactly what what happened. Right? Because when we are I I you know, one of the words that I would like to normally use, you know, back then in when I'm in Angola, and my mother says, oh, the Internet is so slow. The Internet is so poor and blah blah blah. I keep telling them, listen.
A few years back, we we totally would be dark on dark. Right? But now we still can survive. You still can do your own stuff, right, with a lot of issues.
Right? No Internet banking systems or whatever as you mentioned in Nigeria, but you still can send a message over WhatsApp at least, and it will come through with the with the x amount of delays or whatever it is. Right? But I had a very good friend of mine.
He he runs an ISP in DRC in Shasta. And then eventually he told me, Darren, listen. I'm an I'm in a huge problems because, you know, the country is almost dark, and I need to have some connectivity. And whatever he did, you know you know, as a human beings, I think we start getting innovative, you know, and we start looking into, possibilities, whether it's a piano or sax or cell or whatever, just to have your own people online.
Right? And so what he did was funny enough. So he went to he had to deal with, if I remember correctly. And then he went through through Angola and from Angola to Fortaleza.
And from Fortaleza, funny enough, he didn't went to US, but he he went with the link back to Lisbon.
Wow.
Right. So we were looking into the latest perspective, and he said, you know what? Listen. The latency might be better going through North America, going for, from Fortaleza all the way up to Lisbon.
It doesn't really matter because people are online now. That's what I wanted to to make sure about it. Right? So they True.
My customers, at the end of the day, they still can surf over the Internet. And it was funny enough to see that, you know, because at the end of the day, as you mentioned, you know, are are we there already? I don't think we are there already because if we look into the major markets in Africa where a lot of content and, cloud service providers are going to South Africa, Nigeria, and Kenya. Right?
Even them sometimes do face issues when there is a cable cut. Right? So do I like to see something different in the future? Hell, yeah.
I think we all we all would love to see that. Right? But at the end of the day, I think we there is so many autonomous system numbers involved in making all of this happening, you know, that it really takes, you know, an hard bite to make, you know, everybody, you know, first of all, trying to localize. There there is a enabled data center available data center with enough power, you know, to serve the CDNs, and then the CDNs do have a strategy, to serve that specific country or region locally and so on and so forth.
You know? In Angola, eventually, as far as I'm concerned, we only have caching solutions so far. Well, that was better than nothing because back in two thousand eleven, you know, a price per megabit in Angola for the end user used to be seven hundred bucks.
Wow.
Back in two thousand eleven.
And I don't want to do any kind of spoil spoiler over here, but I'm just writing a book about that, you know, about Yeah. Of Internet in Angola. You know? And that's one of the sharpest sharpest I would like to to emphasize because, you know, people and users always tend to say, oh, the price of per bank a bit is too high.
It's too high. It goes lower. Obviously, we cannot compare ourselves, for instance, with Portugal or with with, you know, Ashbourne or whatever the place is because those market we we are talking about macho markets. Right?
So if we eventually on also basis, we went from, I don't know, two fifty, bugs per megabit down to twenty three or down to twenty now, it's an improvement, right, over the last decade, I would say.
Let me add for people who don't speak this language, like, I don't in the United States, like, this is in the in the wholesale market of transit, you you you do you quotes like this of just, like, how much, you pay by volume of how much traffic, and there's usually a a meg month is the is the term, and what do you what's the cost for a meg month? And in the United States, I don't know, like, fifteen years ago across below a dollar, was kinda like the going rate. It's like the gallon of gas kinda metric.
So so we got below a doll. I don't know what it is now. I mean, it's, like, ten cents or something. I don't know. It's it's gonna be something really small in the United States.
But in a lot of developing parts of the world, like Darwin's saying, it's a seven hundred dollars a meg month.
And, I remember hearing like South Pacific Islands pay for satellite and be like fifteen hundred dollars meg month. So it was fifteen hundred times the cost, for the transit. I mean, it's, like, outrageously expensive, and, you know, there's reasons why there's just there there definitely been parts of the world. All these prices are going down all over the world, but, but there's still, like, some there's been some huge disparities, like, of multiple orders of magnitude, like, in this example.
But True.
And all of these, thanks for translating that into a normal language, Doug. I think it's very important for our listeners. But at the end of the day, it all comes down with, you know, how much is is the market. You know? What's the strategy? What's the country's strategy, you know, to becoming even more digital?
And, you know, obviously, you know, when I saw all of that, I told my CEO back then in I told him, hey. Listen. I think we are doing something really wrong as a country. And then he looked at me.
I was very young, and he said, what you talking about it? You know? And I said, yeah. We are sending all the traffic back to Europe and then coming back again.
You know? There is something else we should be doing over here. And then, eventually, we decided to look in, in building, you know, another IXP into the country and trying to, you know, to I called myself back then. I called myself a priest because I went back to all the universities, all these, government institutions that tell them how good peering was and how important it was to localize traffic and so on and so forth.
And eventually, if I look now back now, even when I left, you know, I'm it still it still gives me some Philip, you know, to see the good developments that the country has been doing. And I was surprised, you know, even to see the developments that the country that, like, Congo is also doing. You know? Because we don't tend to look into those markets until we there is an event happening in there, and then someone is showing the slides and showing what what what did we had a decade ago and what do we have now.
You know? And I think this all of this will help us out, you know, as a as a as a continent, to go into the next level, which then means, you know, entrepreneurs will then, instead of, you know, hosting everything outside, they will, you know, maybe utilize or use, local clouds which are available in certain regions and countries, you know, and try to do a fall over. You know? Maybe I can just leave, you know, something that I'm I'm I'm doing on a code in country and then having a plan b somewhere outside.
You know?
But So Darwin, you mentioned, like, the, you know, the price this price of transit, and when it's very high, it's a illiberal market.
And and some of this has to do with, like, the infrastructure you're talking about of, like, what's been constructed of, like, how do you do you even have the the, the technical means to keep the traffic local? But, equally important and maybe there's other ingredients that I'm not thinking of, but there's equally important is usually a, a telecommunications regulator. So the government gets involved as well. And and and usually the role of the regulator, is to take the incumbent, whoever is the dominant player.
Usually, in most countries, in nearly every country at some point, the incumbent was the former state telecom founded and possibly still owned by the government, and every country is on a similar path where they fall of just trying to liberalize the market, trying to privatize this, create competition that improves service, lowers price. And, but in every every country, you have a a situation where the incumbents still, maybe they've been partially or completely privatized, but they still have an outside influence, and and it's kinda not in their favor, to do this traffic localization because it it it helps the, their competitors more than it helps them, even if it does help the the country.
The country.
And so so then they go the then then you need a regulator that comes in and says, hey, incumbent.
You don't have a choice. We're gonna force you to connect domestically, and they're gonna complain, and then, and that that helps create a more vibrant market. And so I think and, you know, aside from, you know, having building the infrastructure, there's also the kinda like this legal, government role, and and it's often a mark of how the strength of reg of a regulator of a country if if the regulator is strong, is able to make the incumbent make all this domestic connectivity, then that's a strong regulator. There's countries that have fairly weak regulators, and they are in the other direction where the incumbent still just runs roughshod over the country at the expense of the, the people in the country regardless of, you know, whether the exchanges were built or not. But, anyway, it's a Sure.
No. I can see I can see my I can see myself in what you just mentioned, you know, because in the beginning, definitely, you know, my the the the two larger networks of the country in Angola were not exchanging traffic with each other. And so though this this traffic has been exchanged in Portugal for whatever reason. Right?
And there is a process that, obviously, it's like you said, Doug, it's not only about building the infrastructure because, obviously, we we you know, with with some knowledge and with some, you know, people which are willing to do so, we're gonna do it anyway. Right? But it's also a part of education. Right?
And so this part of the education was, to me, was one of the most hardest to do, you know, but it was rewarding at the end of the day. You know? When you see, actually, that that they are peering locally, exchanging traffic locally, and no end user traffic is going abroad and then coming back again. You know?
So it just give you gives you a feeling of proudness in a way. But like you said, you know, the regulator should be, you know, should be educated, and the regulator should also, I would say, not only asking questions, but also see also see the benefits for this for for a specific country. Right? And so one of the things I'm fighting one of the things I'm fighting now just very quickly aside of of of of the kicks, I do a lot of volunteering activities, but it's just to, you know, to increase, the I p v six utilization.
And, obviously, you might see everywhere. Right? Everybody will say, oh, but we are very slow into that as a world, as a countries, and blah blah blah. But, eventually, you know, I I think Brazil does a very good job on that, especially the guys from Nick dot p r.
They do have a lot of educational videos on YouTube or, you know, courses about how to deploy I p v six and so on and so forth. And one of the things I was pushing the operators in a in a Telegram group together also with the with the regulator was telling them, listen. We we we shouldn't be behind Gabon, for example, because Gabon Gabon is doing a very good job. And how do I confront them with with with with that?
I confront them with with with data, right, with stats. And if we look into Gabon, which is right next door, they are doing an awesome job. Things are going as it should be. Right?
Maybe on on a lower pace comparing to United States or Brazil or countries in Europe, but, hey, we we do have our own pace. Right? And so, eventually, I was bombarding them and still bombarding them everywhere, Twitter, everywhere I can. Right?
And so, eventually, now I'm seeing, you know, a a development. And Unital, which is the largest mobile operator, they are doing in our deployments at the end users with the, with the with the on premises routers, you know, modems to the end users with IPVC enabled already, which is, you know, which is actually phenomenal. And I I'm really looking forward my, trip to Angola end of the year to meet these guys and tell them, listen. You guys are doing a great job.
You know? And since you guy guys are doing, maybe there are the smaller ones, the medium ones, you'll follow your path. Right? That's what we expect for for everybody.
Yeah. Hey. So, I wanna make sure we have time to talk about the the sax cable too. So Mhmm. I wanna set the scene. So this is back in, I guess, the cable came active in maybe September twenty eighteen, I think.
But in that time, in the couple years leading up to it, there was kind of a race, a bit of a competition between who was gonna build the first cable. There was maybe three projects. Right? There was a SAEX cable that never ended up happening.
There's the Saks cable that Angola cables, led, and Darwin, you were involved in that sale, from the Chinese sale ended up getting was built.
They claimed they had, built the cable first. Although, at the time, I was at let's see. This was, twenty eighteen. I we would have already been dying would have been bought by Oracle, so we're Oracle Internet intelligence. We could see it was very easy for us to confirm when Sachs came active.
We couldn't see any impact for a sale. I'm not sure exactly when that, but it is active now. I'm not sure when that happened. But, from our perspective, Sachs was the first one.
But, Darwin, like, you and I had already been chatting about this, and you sent me a message. And he's like he's like, you should check check your check your stats, and, because I think, you know, the cable's up. And, I've been looking for it because at that point, I had already kind of spotted a few cables, and gotten some, press. It kinda became a a hobby of mine that still is to this day of trying to find the noteworthy cables and see if I can publish the first evidence of their use, starting with the Alba one cable to Cuba.
And, anyway, so you gave me this tip. I went and looked. I was like, oh, we've got you know, it's it's striking. It's striking. We we had measurement servers in Brazil.
We measure to providers in Angola, and all of a sudden, it went from three hundred milliseconds to a hundred or less. I don't remember with the numbers. But, you know, it's no longer going, you know, up to North America, across the North Atlantic, back down. It was a direct path now that didn't exist.
There's no way that can happen without, that wasn't possible prior to the activation. But you you sent me a message then, and you're like I so when I started publishing this, and then you send message, like, you're like, oh, I I I don't think I was supposed to tell you. No. I mean, all is forgiven, and it's a great accomplishment.
Everybody's happy, but, you're like, give me give me some color here. Tell me what happened.
Yeah. I I mean, you know, when you when you are on the edge of the things thing things like those, you know, when you are so involved and make it happen, you know, and when you see, this coming into reality, it's even more, you know, more rerouting. And I think the first thing I did, I and was right on the corner if I'm not mistaken. And I, yes, I pinged I pinged Doug, then I said, hey.
I think we got a case over here. You know? But funny enough, let me let me let me go back in the in time. You know?
So I I was traveling from Luanda to Fortaleza because I told my colleagues in Fortaleza, I want to be the I want to see the evidence from the first package from Africa to Brazil. And eventually, when I arrived at the airport, you know, they were receiving a lot of pressure from the CTO, you know, back then. And the CTO was like, no. No.
No. We need to activate now. And they were saying, no. No. No. We are still doing some configs and blah blah blah, but they were waiting on me.
And then I wait eventually, I arrived at the cable landing station, and everybody, Tari, where were you at? I said, I wasn't on the plane. You know? And then they said, let's do that because the CTO is calling us like hell.
You know? And then, eventually, we did a commit. You know? We saw the first buckets going router to router between the two countries, and it was, like, you know, mind blowing.
It was just like crazy. You know? And, looking into the latency decrease, you know, was just was just something special, I would say. And then, eventually, I pinged dog, and I said, hey.
Listen. Like, we we were both at LACNIC, if I'm not mistaken. We were going to LACNIC. I can't remember Rosario.
It was yeah. So it was right it was just before LACNIC. So we're both going to LACNIC in Rosario, Argentina.
But it was it was like a week prior or something.
Right.
So we break the story or at least, you know, confirm, you know, publish the first evidence of the of the Saks cable, say, you know, as far as we're concerned, this is the first, operational cable across the South Atlantic.
And then we meet up, in Rosario, Argentina, and then we did a did a little lightning talk Right.
Together. Darwin talked about the project, the benefits, and stuff, and then I just went through the, what we could see from an Internet measurement standpoint.
And, anyway, it was a cool Doug, what was the actual difference in latency that you observed?
Yeah.
I'm trying to remember, maybe we can find here, I I, I it's something like, you know, a reduction in, from maybe three hundred to one hundred milliseconds.
Let's see if I can pull it up.
It was in either way, it was more than three. To two hundred milliseconds in latency reduction. Wow.
Because from Mongolia to Fortaleis, it's like sixty three milliseconds. And then if you go down to South Africa, you add thirty more milliseconds, so you'll end up into, ninety something. Right? Depending on where you go, Cape Town or Johannesburg. So yeah.
So it was a nice That's pretty pretty dramatic.
Yeah. Because we We'll put it we'll put it in show notes. We have some graphs of the Yeah. Yeah. Improvement.
I I still have some slides. I don't have a with me in here because right after the talk with Doug, I went to Brazil, during the Brazilian infrastructure week, and we were doing some kind of analyzes on online gamers in South Africa and what the benefits were for them. Right? And so they said, actually, you know, it's actually easy for us to go to Brazil than going all the way up to Europe because we have a reduction on latency. And so for the gamers, it was something, you know, something cool to be, you know, because when you are playing online, you know, it's it's it's always critical, right, that a player doesn't get charged just because of latency. Oh, yeah. Absolutely.
And so yeah. That's, yeah. Maybe we can just stay into into what whatever we have dug on the graphs.
I still have something, and then we can Yeah.
I've got my I I there's links to the slides, and stuff, and, I've got my thanks to the Internet archive, because Oracle deleted all of our stuff. We can, yeah. It was three hundred to three hundred around three hundred milliseconds to a hundred milliseconds. And, again, for people who don't, aren't used to these, you know, speaking in these terms, across the Atlantic is maybe seventy, eighty milliseconds. Across the Pacific Ocean is, like, a hundred milliseconds round trip. So three hundred milliseconds is pretty pretty far, but it would make sense to drop to about a hundred milliseconds to get between Brazil and, Angola, and such a thing was not possible prior to, twenty eighteen.
Yeah. Especially when you because, you know, Doug, a few of your questions might be so okay. What was what was well, I wouldn't speak for others, but to myself, what was one of the motivations that we built this cable and we had a huge investment on that? I mean, obviously, first of all, you know, why should we be dependent on Europe or on US to go to Brazil?
Secondly, we do speak the same language, and we consume a lot of Brazilian contents because, you know, if if even holidays, I'm sitting in Europe now, and if my daughter looks up into a content, Portuguese content, the first thing she gets on YouTube is Brazilian content because those guys, you know, they they just generate content dramatically. Right? So there is no actually, you know, I I would say, you know, visible content in Portuguese from Portugal Portuguese from Mongolia, we have something, you know, which is more music related to it. But from Brazil, it's everything.
I mean, it's music. It's, video games. It's like telenovelas, you know, which our parents and grandparents used to consume a lot. You know?
And, eventually, all of these, you know, got even better, but, obviously, the end users, they they won't notice that, immediately like we do, right, on a on this in this industry.
But, you know, eventually, we we did something good for both countries, I would say, because neither we, neither Brazil had direct connection, and we we we couldn't exchange traffic in a more direct way than we are doing nowadays.
So what's, so what's next? What's your next project for for DE-CIX? What are you working on, in your role now?
Oh, yeah. So, that's a good question. I think you guys might might have seen everything on the news, but I was, I was involved in, yeah, just, you know, trying to open a new market, and we announced that we are going into the biggest market in Latin America, which is Brazil.
So, yeah, Brazil, it will be.
That's what I'm, putting my focus, nowadays into it.
So d kicks is opening an exchange in in Sao Paulo?
Rio. Sao Paulo and Rio. Yes.
Okay.
Yes.
You're gonna try to connect to the, Saks, your your colleagues at Anguilla Cables, or, your ex ex colleagues?
Sure. It's a it's it's an option. Why not? Right?
Sure.
Yeah. Yeah. Yeah. It's an option.
So when when are those slated to be active?
So, yeah, we are working, hard to get everything, at least ready by next year March during capacity LATAM.
So, yeah, we're hoping to push the horses as far as they can go, you know, because as you guys might know or not know, and our audience might also not know. But, yeah, shipping the equipments to Brazil might be a hard thing to do. Yeah. But, yeah, we are we are going, running against the time.
Sure. Awesome.
So, Darwin, how do you see actually, Doug, you as well since your background. How do you see the evolution of the Internet, you know, over this last decade, right, the last ten years since, you know, maybe twenty fourteen, twenty fifteen?
And, yeah. Yeah. Like, how do you see trends and technologies changing?
No. Oh, good. You know, I think at the end of the day, we all have different perspectives. Right?
But, you know, everything we do in our in our industry, I always like to really correlate that into whatever we do in our personal lives. And, you know, again, you know, I I think I said that before, but the way we build the Internet, at least from my knowledge and at least for what from what I can see what is coming, towards us, is that we build an Internet for a single propose, which was exchanging traffic among different networks, different autonomous system numbers. Right? And, eventually, we wanted to make sure that the packets were delivered and were sent.
Nowadays, we are talking about those, you know, new generation, those new kits doing a lot of stuff over the Internet. Right?
This Internet that has been built with certain standards and certain protocols, right, which nowadays, maybe this younger generation, they don't give care about it. But it is always good to to to to mention that whenever we have the the chance to do so. But when we speak about, you know, artificial intelligence, you know, crypto mining, all of those stuffs, you know, and then and then you tend to realize when you are sitting on a couch and have your coffee or whatever, your beer, you know, during the weekend, and then if you just realize how how much power we will need to serve all of that, you know, How much how many queries, on ChatGPT are being made on a daily basis or on a hour basis.
You know? Then I I tend to ask myself the question, can we handle all of that? Will we we be able to handle all of that? Maybe, yes.
Maybe we'll. But as Doug mentioned before as well, you know, we are talking about different maturity in different markets. You know? In Europe and US, we we nowadays, we only talk about intelligence artificial intelligence and so on and so forth.
But in Africa or parts of South America, we're still struggling with basic basic connectivities.
Right? That's what that's why we also see projects like Starlink going down and trying to increase the Internet penetration of those countries because the fiber is not, on their doors. And if the fiber is on their doors, they cannot afford to pay it. Either they need to choose whether they pay an Internet connectivity or either they buy food.
Right? So we are talking here about different scales, which I normally like just to bring up, you know, for our own sanity because many times we tend to forget. Right? Everybody everything is going so quickly that we tend to forget, you know, that basic things also also needs to still tackled.
You know? Then I I will hand over to to Doug to to his thoughts about.
Yeah. I guess, I would mention I I would I would think there's, like, two things. So, you know, again, focusing on the continent of Africa, and some of this relates to other places as well. But, you know, in this in this time, in the fifteen plus years, now we have lots of submarine cables instead of just zero, then one or two, and now we've got multiple going down each coast.
So that's obviously a net positive.
What is kinda interesting though, and I think we've maybe not so foreseen, was the fact that, nowadays when there's a cable cut, it's a multi cable cut. You know, like, it's not we put in more cables, and now when there's an instant, it's more cables that are broken. The part part of this has to do with geological stuff. So there's this, you know, Congo Canyon that, has these landslides that go, you know, a hundred kilometers or something, outrageously, you know, long.
They have multiple cables. It's very hard to engineer around some of this, when you have an, incident on the sea floor that's gonna span, you know, enormous distances, and the the the incident in, March, you know, off, you know, in West Africa was another one where you have multiple cables, four cables getting broken in a single incident. So, it does it does, yeah. I'm talking to a reporter from New York Times.
He's working on a story on this of, just, you know, more cables does sometimes equate to more cables getting cut when there's a incident such as one of these. So you doesn't, you know, it doesn't necessarily buy you, you know, completely solve the problem.
You've you've you've Right.
And there's only so much geographic, you know, diversity. Equiano got some, maybe by accident, just being a little farther away from, the problems of both the Congo Canyon and the Ivory Coast incidents.
But, anyway, it's interesting these, you know, more cables equals more cables getting cut, things. So, I it's a net positive, but there's it's not without its snags. And then with the and the other dimension, like the local hosting, one thing, you know, back to, you know, the the question about, like, these local services that were, you know, not working even though you had, you know, you some of the stuff ought to be able to work completely locally. There's another phenomenon that I wasn't aware of until I met some folks in South Africa from South Africa. I was at Apricot in Thailand in February, and we were chatting because they were had been another submarine cable cut.
And there you know, the four folks down in South Africa, I guess, Angola is a little bit in this boat as well where, every time there's a cable cut, there's a that's a long distance back to Right. Europe. And if anything breaks along the way, they suffer. Like, they just, and there may be lots of cables.
And when multiple cables break, they, they they struggle, to, maintain the quality of service. I mean, they're staying online, but it's it's a matter of quality of service. But one of the phenomenon that's interesting that I didn't know about was that, because I'm we were having this conversation. Well, what about all this localization?
You have this local content, local like, can't you just get your, you know, Netflix from the local server now? I get your you know? And, you know, like, you would think so, but there's a a lot of times when it's happening is, this local so say it's, you know, a CDN, pick your CDN, it's got its local cache of all this content.
Those caches, if they, don't have a great connection back to home base, because, say, there's a submarine cable cut, they may drain themselves and just take themselves out of operation, which has which kinda compounds the fat the problem. So in South Africa, there's content local, but it's being taken out of service because the local content, service, like CDN, nodes can't get a quality connection back to home base. So they just take themselves out of service. So then all these folks in South Africa now have to use, instead of using a local comment, they are pushed onto the the cable, cable rings that are now, you know, having problems.
And it's a it's a it's a weird, thing. So we've we've, we've made a lot of progress and things have gotten a lot better. But when we run into some rough patch, it doesn't and we there's some problems still to solve, there where, in both these, you know, the the summary cable connectivity, local hosting, local caching stuff, each one, it's a net positive. We've grown and evolved, but there's, there's a there's ours are a lot of challenges still, to try to, you know, maintain, quality service when you have some, you know, catastrophic technical failures.
Yeah. And then again, I think this is a this is this might be a global thing. Right? You you mentioned a couple of times the word phenomen, right, happening in our industry.
We were chatting about it, about the Netflix fight, Jack Paul and Mike Tyson. And we we I was suffering while I in Europe, right, just to get a a good connectivity to watch the to watch the fight the last March. But in Angola, they were watching that fight without any issues, and they were they were, you know, making fun of me. And I was like, okay. So there there is something wrong going on here. Right?
So You know what? I think I think in that case, so we're talking about the the was it, Jake Paul, Mike Tyson, kind of ridiculous spectacle fight. I I tried to watch some of this too. It wasn't working so well for me, but it it seemed like it really came down to, each provider, how you connected to Netflix.
And Yeah. Netflix.
Netflix is just a single entity that only they have their own CDN that they exclusively use.
That's you know, one of the theories floated of why people are having problems, was per service provider, if if you didn't have a lot of ways to connect to Netflix or if you were trying to squeeze too much traffic through, a constrained link, then you would have problems. Some supervisors had had very few problems. Some had a lot of problems, but it, but it did seem like the, the fact that there's it's just this one CDN as opposed to most events, content events these days are split across, multiple, content providers. Even when prime Amazon Prime Video does Thursday night football, you think, oh, it must just be all AWS, and it's not the case.
They they split that against competing, content, providers or cloud providers, to, spread out the load. And I don't know. I guess that too occasionally suffers from its its glitches, but at least it's on. So that was one unique aspect of, the Netflix case.
But that's that's, that's wild that the folks in Angola were watching that with no problem.
I I'm glad they gave you some ribbing for that.
It's true.
Yeah. No. I you know, just just to finish to sum up, you know, I think, like like all Doug also mentioned, you know, it it comes up with, you know, so many factors, so many analyzes, so many stakeholders, you know, that eventually, whenever we think we are safe, you know, we might not be. You know?
And, you know, and going back into this incident from March twenty four, I think, you know, those cables that were impacted, you know, and and from Google was not. You know, I I think it's also important for the guys who are building those cables, you know, to kind of doing a different desk desktop survey than the ones who are already laid down. Yeah. Right?
And maybe just try to avoid that for the future. That's that's one thing I could think of as well. You know?
So, Darwin, you gave your thoughts about the evolution of Internet. I gave a couple about Submarine Cable as a content provider. So would you how how would you describe, DE-CIX' role in this, you know, the evolution of the Internet?
And what role does it play?
Yeah. I think, you know, as as we all know, you know, an Internet exchange point is, is a crucial component of our of our industry. Right?
Because we need to make sure that we can, you know, we can, have networks connected. They they can exchange traffic in a secure way and faster way. And And I think what we are doing at, you know, it's, it's not obviously trying to go everywhere, which doesn't make sense. But one of the things we have on our DNA is if we go somewhere is really to try to solve a problem. Right? A problem that might might have been identified by external or internal stakeholders.
And we always try to, you know, just to to have a different, conversations with the local community to understand, hey. Does it really make sense for us to come in, or can we work together with the with the with the with the op with IXP, local IXP to make things better like we did in Switzerland, right, where we have a part partnership with SwissIX, where we do the technical part and they do the business strategy and so on and so forth. But it all comes down into which problem are we trying to solve as a company. Right?
And if we can do that, you know, on a on a a medium large scale, I think this is beneficial for each and every network connected to our platform and also at the end of the day for the end users because they at the end of the day, we all end users. Right? When we are sitting on our couches at home with the family and enjoying something on, on, on the web. Right?
And so that's that's what we at DeepKick normally do. Right? And, for example, you might ask, so why Brazil?
Eventually, you know, Brazil was a market which took us a little bit, longer time because we needed to really to do a a deep dive and assessment, because IXBR is doing a tremendous good job as being the largest Internet exchange point in the world. But, obviously, you know, there there are kind, kind of components where we can aggregate some value because, on the at the kicks, we don't focus ourselves only on pairing, but we do have more than just pairing. Right? Like, closed user groups, black calling on route service or, you know, enterprise, networks which are coming on board. And, again, it's, it's all about educating. Right?
We we now tend to help those enterprise networks to come on board and also to be able to speak BGP because most of them, they don't do that. Right? And, eventually, we started all of, with this in in Germani. And what we trying to do is trying to replicate this and try to educate these enterprise guys what the benefits of pairing or of an IXP might be for them and for the business at the end of the day.
Gentlemen, it's been a a real pleasure to have you, Doug, as usual. And, Darwin, getting to to hear some of your history, especially, of course, your personal history and your experience on professional sport. That was really interesting. But but, of course, working in the submarine telecom cable space, which is something that I've been interested in over the past few years here at work at Kentik as well. So thanks so much for joining today.
Thank you, guys. I really appreciate. And, yeah, looking forward to to the podcast when it comes available.
Absolutely.
So for now, if you are interested in coming on Telemetry Now as a guest or or if you have a idea for a podcast, I'd love to hear from you. Please reach out at telemetrynow@kentik.com. So for now, thanks so much for listening. Bye bye.