______________________________________________________________________ DRAFT TRANSCRIPT SIG: IX Date: Thursday 2 March 2006 Time: 2.00pm Presentation: The routing aggregation policy: a failed social experiment at the LINX Presenter: Nigel Titley ______________________________________________________________________ PHILIP SMITH: I think we should make a start. Good afternoon, everyone, and welcome to the special interest group meeting. We have two sessions for you this afternoon. The first session is going to be full of longer presentations. The second session we'll hear mostly from the Internet Exchange Point updates, within the Asia Pacific region and around the world. Now, before we start, we've got some housekeeping. If you have any questions for any of the presenters, I'd like to ask you if you could please use the microphone. Use it and don't just stand somewhere close to it because we can't really hear what's happening. Also this session is actually broadcast on the net. So there will be a lot of remote participants, so that they can actually hear what's going on as well. When you ask a question, please state your name and affiliation just again to help with identifying who you are and to help with the stenographers. OK, afternoon tea area - you probably worked out where it is now. It's over there to my left, in around the two sides of the Convention Centre. There's the APRICOT closing event tonight at 7am in the ballroom foyer. It's at the end of the lobby area, up the front of the ballroom. APNIC would like to remind you about the MyAPNIC and the policy flash demo which is running all day at the APNIC help desk. The help desk is available during the breaks. Also, check the onsite notice board. But you just check the onsite notice board on the APNIC website just for any late changes. And tomorrow there's the APNIC 21 informal closing dinner. It says it's informal but it seems to be getting more formal with more reminders and a bigger invitation list and so forth. If you have completed the ticket payment, please come and collect your ticket from the help desk today. So basically from now and from tomorrow. If you don't know about it, there are more details on the APNIC 21 website. Before we launch into the presentations, just a little bit of administrivia. Two chairs - Che-Hoo Cheng and myself, Philip Smith. You can reach us at sig-ixchair@apnic. The agenda for the first 90-minute session. The special interest group had no action items, so we can skip over that part of the agenda very quickly, and that takes us into the first of the four presentations/activities. The first presentation is from Nigel Titley who will be talking about the routing aggregation policy. Or more likely the failed social experiment at the LINX. NIGEL TITLEY: This will be a fairly short presentation. When Philip heard about what we were trying to do at LINX a year ago, he invited us along to make a presentation. It was expected to be a long presentation with lots of drafts showing how we affected the aggregation on the Internet and how everything was so much better. Things didn't quite work out that way. OK, for those of you that don't know, the LINX is the London Internet Exchange, used to stand for the London Internet Neutral Exchange, somewhere along the line we lost our neutrality. About 90 gigs of peak traffic. We're a distributive exchange. We have seven locations in London. We are Ethernet based. We were found in 1994. We're one of the oldest Internet exchanges as well. To understand this presentation you need to understand a bit how the LINX is organised. It's a company limited by guarantee, which is interesting, really. English law doesn't have the certain entity as an association, so you have to be a Limited company. Each member of the LINX owns a share. If the LINX goes under, you lose a pound. We're governed by two documents. One is the Articles of Association which is a very, very difficult change. And one is the Memorandum of Understanding which is easier to change, which governs how we run the LINX and governs the rules under which the members must exist. And the sort of things they have to do to remain to be a member of the exchange. It's owned jointly by members and all members must comply with the Memorandum of Understanding otherwise they're liable to be thrown out and this has happened on one or two occasions. It's threatened quite often. We have thrown somebody out for one point for routing their traffic through other members without asking. The company went bust shortly afterwards. They were out of the country before we were able to serve them with their breach notice. Changes are voted on by the members. At annual general meetings and overall the things are run by a board and staff. OK. Back in 1994, the Internet was still a nice place, and the LINX founders had a sense of social responsibility and in particular there are two social responsibility clauses. One is an all routing policy, routing policy. It must be registered in a public routing registry. That's something like RIPE or APNIC or whatever. The other statement was that route should be aggregated as far as possible. Now, aggregated as far as possible? It's a terribly warm, fuzzy statement. And at the time it sounded good and everybody sort of understood it. But it's really not very enforceable. Furthermore, we actually only checked it when people joined. And nobody checked it afterwards. There was a feeling that either we should get rid of this statement altogether or we should actually make it enforceable, measurable and enforceable and regularly checked. Here's the reason why we have an aggregation policy - it shows the number of prefixes on the Internet and everyone knows the report comes from the CIDR report run by Geoff Huston. It shows the extent of the problem. Apart from 2001ish, which we all remember, prefixes have been increasing steadily. People don't aggregate properly, that's one of the reasons they are increasing. The pink line is what we should be seeing and the yellow line is what we are seeing which is a bit sad. Back to the story. When we have to change policy we always go out and ask the members. What normally happens is we ask the members and ask them to pass the resolution and we go and do it and put it into the Memorandum of Understanding. LINX meeting 48 passed a resolution to adopt a mandatory enforceable route aggregation policy. It was a fairly close vote - 19 to 17 - but nonetheless it was a mandate. And the council got charged with coming up with an algorithm of enforcement. We duly went away and did. From the ISPs, autonomous system, should be less than Nx2+3. Where N is the minimum number of blocks they could possibly advertise. So we multiplied it by two to make it. We gave three on, so that small ISPs could have a bit more slack. And at least it's measurable. And we were proposed to reduce this in 2006 to Nx1.5+3. But anyway. OK. We now have the rules. Somebody went away and wrote a script to look at the number of routes people were advertising and produce a sorted list of sinning members and we notified, identified the members that needed to take action, the ones who would be in breach if this rule is actually voted in. And we gave a fair amount of time for this action and we started discussions with members. We prodded them and said, "Do you realise you would be in breach?" The other thing we noted was that you would need to make provision for an emergency situation and we just apply common sense. And then the idea was that at LINX quarterly meetings we would publish the list and everybody would cringe in their seats and go away and fix the networks. And this works quite well, actually, it's surprising. So by December '04, we had this thing in place for two or three months, and we saw some substantial improvements. Examples here - UPC was originally 8th on the list and they moved down to 63rd, well inside the breach. Indeed. COLT, Entanet, Rednet, Solnet, all moved to 0% which was very good indeed. Things could be changed was what was proved. And the members who didn't actually exceed the two times metric did say, "We're going to fix it." And it seemed to be working. One or two large ISPs said, "We have huge blocks of IP addresses, we don't want to announce the whole /14 or whatever, would it be OK to announce /16s?" We agreed anything shorter than a /15 could be broken up into /16s and that was fine. The other thing apart from certain breach notices, people suggested that maybe making good route aggregation could be a requirement of transit contracts. If you were buying, if you were buying transit from somebody upstream, you might say, "You're on breach of the LINX regulations, we won't buy from you." We thought, maybe. And to facilitate this we put together a BCP and produced some standardised wording for people to have purchase contracts. All sort of gentle nudges. And other people suggested maybe good route aggregation was a peering agreement requirement. And furthermore, we sort of spread the word a bit. Euro-IX gave us a platform. Most people seemed to understand. We made the announcement at RIPE 50, only arranged to make the presentation at APRICOT '06, that just shows how successful everything has been. RIPE also has a note that says aggregation is the responsibility of the network operators. But again they encourage to aggregate as far as possible. What happened? Many members actually improved their aggregation. What came out of this actually was it was very, very surprising that most members didn't even know how badly aggregated they were. That they were advertising /24s and all sorts of crap all over the shop and they didn't realise it. So providing a metric to show they were doing this was actually quite a good thing. And they went away and fixed the router configs and a lot of it got better. This is where the story starts to take a bad turn. We started to get member backlash. And it was mainly from large, badly aggregated members who I have to say it were mostly US-based. And they decided that an Internet exchange had no rights to be telling them how badly aggregated they were. And they basically organised a member backlash. Remember, at this stage, nothing had actually been voted into the Memorandum of Understanding. Anyway, we got as far as LINX 50. The opposition managed to get the mandatory aggregation policy thrown out. So we had no mandatory aggregation policy. It was converted to a BCP. And we do have one you can look at that makes recommendations as to how you should improve your aggregation. In one sense, the mandatory aggregation policy was a failure. It would have been nice to improve the general aggregation across the world. There have been some successes. Some LINX members will, even to this day, still only appear if you meet the BCP. It did raise the profile of aggregation amongst the members. And made them aware of how badly aggregated a lot of them were. And I say an online check is available - it will be available shortly - and it will tell you whether any particular AS meets the BCP. And that's the end of the story. NIGEL TITLEY: Because it's not particularly easy to see how badly aggregated you are, they just are unnoticed. My employer was in the top five and it was, you know, purely the fact that we had one or two routers that were misconfigured. SPEAKER FROM THE FLOOR: Merely by notification and finding the right people and continuing down that path, without using any strongarm tactics? NIGEL TITLEY: That was what we hoped. We did see that indeed. But I mean, the CIDR report has been going out for so many years. RANDY BUSH: That's the problem. They look at it. And if you are - sorry, Randy Bush, IIJ, also from the US, that's what the J stands for. You write to people in the CIDR report and they say, "We know it. It's intentional." I would call it a success, not a failure. You succeeded in changing it by routers. You can keep your policy. NIGEL TITLEY: How long will that remain? RANDY BUSH: Maybe it's a process, not a policy. NIGEL TITLEY: At least we have a BCP and you can measure it. It's likely to go the same route as the CIDR report. RANDY BUSH: The measurement tool might be interesting if it could be generalised - show me what I'm doing. NIGEL TITLEY: It does pull it out of... RANDY BUSH: Even if I'm not a LINX member? NIGEL TITLEY: It will be generally available. The CIDR tool does this as well as far as I'm aware. RANDY BUSH: Give them a website they can go to and play. NIGEL TITLEY: Maybe there's some success out there. Basically, we're actually only aiming that ISPs own routes. Not the customers at all. We decide that was well outside anything we could possibly do. At least to start with anyway. SPEAKER FROM THE FLOOR: As the topic of the day seems to be traffic engineering, I had to ask, there is this old mythical concept as long as you're a a peer, you can select which router you want to have. It's much harder. So, do you think that the reason why you were still, because these are peering sessions as far as you know, you can't guarantee. NIGEL TITLEY: You're still allowed. SPEAKER FROM THE FLOOR: Do you see the people that did as part of transit sessions or traffic engineering or therefore refused to do this or the people who didn't do this, was it purely because of something else? NIGEL TITLEY: We never really got to the bottom of the few that wouldn't aggregate. Not all of them were American, I have to say. The opposition was organised largely by transit companies. I won't mention the one in particular. But it wasn't just US. There are a number of Asian carriers who are quite badly aggregated as well. We made no headway at all. SPEAKER FROM THE FLOOR: The carriers are not local to Europe, therefore might be more able to do it - with less issues? NIGEL TITLEY: That is entirely possible. PHILIP SMITH: Any other questions for Nigel? If not, thank you very much. APPLAUSE