______________________________________________________________________ DRAFT TRANSCRIPT DNS operations SIG Thursday 24 February 2005 4.00pm ______________________________________________________________________ JOAO DAMAS OK, let's get started with this SIG, the DNS Operations SIG. It's already past four o'clock and the agenda today is quite full. To start off, I am Joao Damas, I am not Joe Abley, the chair of this group. I am chairing on his behalf. He could not attend. He had some family issues. Nevertheless, he sent a note thanking the APNIC staff for their involvement in preparation work and support as well as managing the technology to allow for remote participation and I'd like to thank Joe in thanking the APNIC staff for that. On the screen is today's agenda. As you see, it's quite full so we can go through that. The review of open action items for previous meetings. As far as I know, there is only one - action dns-16-09-01, the APNIC Secretariat to implement proposal "lame delegation cleanup revised". This status of this is done. Proposal was implemented on 30 September 2004 and Terry Manderson from APNIC will talk about this later in the session. Geoff? GEOFF HUSTON Good afternoon, I'm Geoff Huston from APNIC. I'm here reporting on some work that is happening actually inside the context of the IETF and in particular in the context of some work that's associated with the Internet Engineering Steering Group and a policy proposal that's been put inside the IETF but it certainly impacts upon APNIC and impacts upon folk in this room here. Its all about a reverse DNS space for IPv6. When IPv6 was first specified - maybe not first, it took a little while - the equivalent of the v4 in-addr.arpa space that IPv4 addresses and resolves them through PTR records into fully qualified domain names, you couldn't do it because the in-addr.arpa was already taken so the proposal at the time was to adapt to new ways of thinking. And the idea was to set up an ip6.int and occupy that with reversings. This was originally identified when ip6.int was identified. Life moved on and we all learned and one of the changes was to put all the address and routing protocol work into the arpa domain and in RFC 315 2, which was published in August 2001, that document said that they would deprecate the use of ip6.int in the standard specification and replace it with the use of ip6.arpa. Let me quickly just read exactly what that meant because it was funny wording. "In this context 'deprecate' means that the old usage is not appropriate for new implementations and ip6.int will likely be phased out in an orderly fashion." IPv6 can be phased out as well but ip6.int can be phased out in an orderly fashion. It didn't actually say, "Don't use it any more." And now almost 3.5 years later there's still some use of ip6.int and they're both active. So ip6.int has some mapping domains, ip6.arpa has mapping domains. The observation is that most folk do indeed use ip6.arpa and ip6.int contains only a subset of that reverse space. And it's also been pointed out, particularly by the folk who are operating the delegations for the RIR address allocations that having two active reverse domains is unhelpful, its confusing for the registries, it's confusing for Internet service providers, it's confusing for venders of software. It's confusing for everyone and last and not least, and certainly importantly, it's confusing for end users. What do you look up? What does it mean? So when the Internet Architecture Board considered it, they thought there should be some work in the IETF to complete the phase-out that was promised in RFC 3152 and a draft has been written by an advisory group to the Internet Architecture Board and this has been tentatively sent to the IESG to say "let's stop operating ip6.int". Now, it doesn't quite say that. The text of the draft, draft-huston-ipv6-01.txt from memory says, as of 1 June 2005, in other words, pretty quickly, don't make this formally part of the IPv6 standard. So, if we're supporting - and the registries do support - a standard compliant IPv6 operation, there is no requirement for them to support any further delegation to the ip6.int sub domain as of that day. That's the proposal. And the text of that also asks the RIRs to work with their communities and in particular, in this case, APNIC, to adopt the schedule of stopping support for ip6.int delegations and it might well be that 1 June 2005 might be a good date - it might be a bit aggressive, it might not - but that's the essence of the proposal. And that's it. Previously when we have discussed this, it certainly raised some controversy, particularly with those folk involved in the administration of ip6.int. I don't know if there are folk with questions or comments about it this time and if there's any advice that you would like to offer APNIC regarding deprecation of their current support for delegations in ip6.int. Now would be a good time to come to the microphone and offer those observations or suggestions or comments. ED LEWIS I have a question about the date. 6bone is being terminated I guess around June 6 a year later. Do you want to tie that to this? Do you think it's worth a tie? GEOFF HUSTON Considering the level of confusion it's currently causing and considering the fact that the ip6.int domain has only a small subset of all delegations, it didn't seem worth making it live for any longer than it need live. And certainly there was some discussion about whether today might be a better date than 1 June 2005 and we thought, well, we'd give a little bit of notice and maybe 1 June is the right date. The proposal was quickly, not even within a year but quickly. ED LEWIS I thought ip6.int was also space that's in the 3FFE space, not in the .arpa space. That's not exactly the subset. GEOFF HUSTON There's no reason why it cannot be. You're sure it's not in IPv6 - ED LEWIS I'm sure it's not there now. I'm not here to debate that at all. I think the space is only in ip6.int. They have a year and six more days till they're going to turn that down. That's why I was questioning the date. GEOFF HUSTON Fine question. JOAO DAMAS The fact that it stops being a part of IPv6 standard, does that mean you have to drop it at that point? It just means you don't need to support if? You are free to do what you choose but we are no longer required to do so? I'm wondering whether it would be an appropriate action here to ask APNIC to come up with a plan for the operation of the IPv6 - GEOFF HUSTON >From the point of view of APNIC, it doesn't operate or manage operations in the 3FFE Space. APNIC could say that, for the address space they administer or look after, which is drown from 2001 /16 and 2003 /16, they could take the action earlier or later. It makes no difference. About ip6.int itself and it's continued existence, the fact that there is 3FFE occupying that and will occupy it until June 6 2006 doesn't actually have an awful lot of impact on that. It could continue on until that date quite easily and not be inconsistent with this. IZUMI OKUTANI Izumi from JPNIC. This might be a bit too operational but I'm concerned about the LIRs under our management. And for those who have registered at this current moment, not the ones who are going to register int in the future, what would happen to them? Do they have to delete the int registration and move to arpa? GEOFF HUSTON Good sense would say that, if you haven't also occupied in ip6.arpa, you should do so right now and that would be a fine recommendation. So if there's transfers pending, it would be good to advise them that realistically ip6.int is going away, no matter what, and implementations are swinging to use ip6.arpa as the point where they do the reverse delegation resolution. So that advice ends. When folk want to do new delegations in ip6.int, maybe the advice is, "Why are you doing this? It's not part of the standard conforming IPv6 any more to look in that space. It's only partially populated anyway. Maybe the cost of running two separate nameservers and spaces for this is just not worth the effort." IZUMI OKUTANI OK. Thank you. JOAO DAMAS I think there are staff from all the four RIRs here. I'd like to ask each of them whether it is the case that they are not registering 2001 space address delegations into ip6.int space or not. RICARDO PATARA I am Ricardo Patara from LACNIC. We do both and some of our request delegations delegate both in arpa and int. It would be easy. OLAF KOLKMAN Olaf Kolkman, RIPE NCC. I think we do both but I'm just not sure. I'd have to check. RICHARD JIMMERSON Richard from ARIN. I think that up until recently we were still doing both but I'd have to check to be sure. TERRY MANDERSON Terry Manderson from APNIC. We still support both. JOAO DAMAS Next speaker this afternoon is Fujiwara-san from JPRS. KAZUNORI FUJIWARA Hello. I'm Kazunori Fujiwara from JPRS. Today, I talk about improving reverse continues lookup performance, glue in reverse continues. Today, I have three topics - glueless issues in .JP, the current reverse continues situation and improving reverse continues lookup performance. I define two terms - in-bailiwick nameserver - FQDN with their domain to nameserver. In this case, glue is necessary in delegation. And out-of-bailiwick nameserver - FQ DN with outside domain nameserver. JPRS changed the way of handling glue in July 2004. This change is described in RFC policy 2181, section 6.1. All of out-of-bailiwick glues were deleted. As a result, glueless delegations are increased. But this causes a problem. Some domains make difficulty of name resolution. It includes one of the most famous websites in Japan. We found a problem in BIND8 cache server behaviour about glueless delegation processing. Bind 8 caching nameserver behaves in iterative query, BIND 8 starts nameserver host name resolution at glueless delegation. But, if all nameservers are glueless and all IP addresses are unknown - not in cache - BIND 8 stops first iterative query and does not answer anything. After timeout, stub resolver or application retries querying to caching server. Before then, glueless nameserver addresses may be in cache. As a result, name resolution becomes slower for waiting timeouts - 5 to 30 seconds. At the second time continues query and after that continues query, the DNS cache works well. Therefore, the problem has been hided. Bind 8.2 caching nameserver has more worse behaviour. BIND 8, up to version 8.2.7 - name resolution fails when glueless delegations twice continuously. And other caching nameservers implementation - BIND 9, dnscache and Windows continues service has no issues of gluelessness. We reported a problem found in. JP at NANOG33 meeting and we got a response from Paul Vixie at ISC. This is a problem caused by older BIND 8 and BIND 4. All known workarounds involve providing more glue. In-bailiwick glue is an appropriate workaround. In current reverse continues, RFC 1912, section 2.3, glue A records, it is described, "You shouldn't have any A records in an in-addr.arpa zone file." and current RIRs, NIRs, LIR s reverse delegations, reverse registration is limited to FQDN which is outside of in-addr.arpa zone. As a result, reverse continues lookup is always glueless. Why is reverse continues lookup slow? In many cases, reverse continues lookup is slower than standard continues lookup. The lame delegation is thought of as the most popular cause of this. But glueless delegation is certainly one of the biggest causes of this slow continues lookup. Most of nameservers in arpa zone are out-of-bailiwick names and this causes gluelessness. BIND 9 cache severer can make reverse continues lookup much faster than BIND 8 cache server in most cases. Next - I will explain one continues BIND continues lookup. I look up 202.11.16.167. This is a company's web server. Plus query 167.16.11.202.int.arpa PTR to rootservers. And rootservers answered 202.in-addr.arpa nameservers - there are six servers and rootservers - seven rootservers answered with only NS.RIP.NET glue. Almost all of the nameserver IP addresses are unknown. This glue is special. Only BIND 8 rootservers answer and NS.RIPE.NET is outside of 202.in-addr.arpa zone. It is ignored. And then BIND 8 starts nameserver address resolution and stops resolving. This causes a client timeout. Our next - look up nameservers IP address from rootservers. In this case, look up NS1.APNIC.NET rootservers. In this we look up NS1.APNIC.NET A to.NET servers. In this, the answer is APNIC.NET serve nameservers with glue. The query NS 1.APNIC.NET A to APNIC.NET servers. And the answer is NS 1.APNIC.NET IT address. And it answers - then 202 in-addr.arpa servers address is resolved. Then return to 167.16.11.202.in-addr.arpa PTR lookup. This query, query 167.16.11.202. in-addr.arpa PTR to NS 1.APNIC.NET and the answer is 11.202.in-addr.arpa nameservers. There are six nameservers and all nameservers' IP addresses are unknown. Then BIND 8 starts nameserver a resolution and stops resolving. Then the cause of client timeout again. Next - look up nameservers' IP address from rootservers. In this example, lookup a.dns.jp address. Query a.dns.jp A to rootservers. Rootserver answers.JP nameservers with glue. Query a.dns.jp A to.JP servers..JP server answers a.dns.jp IP address. Next - return to 167.16.11.202.in-addr.arpa PTR lookup. Eight - query 167.16.11.202.in-addr.arpa PTR to a.dns.jp and it answers two jps nameserver names. And all the nameserver addresses are unknown. Next - look up nameservers' IP addresses - ns 01.jprs.co.jp to.JP servers..JP server have been cached at 4. Then query ns 01.jprs.co.jp to.JP servers. It answers JPRS name searchers with glue. In this case, ns 01.jprs.co.jp address is added as glue. Then we should query ns 01.jprs.co.jp is glued. Then 167.16.11.202.in-addr.arpa PTR lookup. 10 query, query 167.16.11.202.in-addr.arpa PTR to ns01.jprs.co.jp and it answers final in PTR jprs.jp. As a result, this reverse continues lookup requires 10 authoritative server queries and in BIND 8 case, three client timeouts occur. In "dig" case, default timeout in 5 seconds. CIDR delegation needs more queries and real cache server includes more IP addresses. Next - improving reverse continues lookup performance. To avoid glueless delegation, my recommendations - to use in-bailiwick nameserver in in-addr.arpa and ip6.arpa and add glue information to reverse continues. For example, 202.11.16.0/24 case - A.NS.16.11.202.in-addr.arpa and B.NS.16.11.202.in-addr.arpa domain's nameserver and two nameserver address glue. All delegations have effective glue. Reverse continues lookup finishes three or four queries in most cases and there are no timeouts in BIND 8. 202.11.16.167 case, only four queries. Plus, rootserver answers APNIC server with glue. Second - APNIC server answers JPNIC server with glue. And third - JPNIC server answers JPRS server with glue. Fourth - JPRS server answers PTR. Saves cache server cost and decreases resolving time. In-bailiwick nameserver's benefits - decreasing resolving cost and decreasing resolving time. Using in- bailiwick nameservers removes a dependency of TLD is' DNS tree only depends on rootservers IP addresses. This requires some changes. In registration system to accept in-bailiwick nameservers and to accept glue A/AAAA and re verse continues registration policy and user's continues configuration change. And another .arpa case in ENUM. In using in-bail wick nameserver on e 164.arpa zone is useful. Finished. JIM REID I think what you said there is very sensible and we should avoid those delegations. However, I think there's a potential problem and that is that, because the delegation is done, it's always been delegated to somebody and they're free to change DNS records and one of the problems you always find with the DNS is that the parent delegation information ends up being radically different from the information that's in the trial zone. So do you have any plans for doing sanity checks to correlate the difference between the parent delegation with what the child has with continues records. YASUHIRO MORISHITA So you need some kind of site to check that the effective DNS records in child zone? JIM REID It's making sure the parent-child zones are in line with each other. The child can change the DNS records but forget to inform the parent. YASUHIRO MORISHITA So this is the same case as the TLDS case. In ip6.arpa zone currently has two effective glue items. Is that correct? OLAF KOLKMAN I don't see how that differs from today, Jim. That goes out of sync as well. Jim is not listening. What is the different situation from today, where people configure nameservers in a directory that go out of sync as well. JIM REID There's absolutely no difference. That's precisely my point. Problem is that, as it - it's all well and good for a parent zone to say to the child to inform space and reverse space to say, "Please make sure when you change your records inform us as the parent so we can keep your delegation correct." That's all well and good. Unless you can find some way of checking that and at some point coming back to the child and saying, "Hey, your records are out of sync. Could we get it updated and corrected?" That's all I was asking. Rather than just have a recommendation, I'm proposing we have some way of checking and come back and say, There's a discrepancy, please sort it out. While you were speaking at the mic, Ed was asking - the chances are it may not be a lame delegation. You could have a scenario where there was one NS record in the parent zone, which matches with what's in the children. Eventually things appear to be working because there's one server in the parent that is referring to an active zone server for the child. JOAO DAMAS The process - you have to implement to police that sort of is similar to check the lame delegation as well. I suggest we wait for Terry's presentation. ED LEWIS I was going to add that this does make the registry responsible for doing lame delegation checking but I think most of the RIRs - I forget about RIPE but the others do - have a lame delegation policy. JOAO DAMAS This actually affects not only RIRs. It affects NIRs, LIRs, as well. Maybe we should think about it. One of the conditions is actually to introduce something in the registry - I don't know if anyone has a position on whether they would consider doing this or not and why. DOUG BARTON Doug Barton from ICANN and IANA. I approach the topic and microphone with a great deal of hesitation. I have two hats in this area. One is as a former DNS administrator for a relatively well-known Internet company. My gut feeling is that, in glue in lots of places throughout the tree is probable particular for all the reasons that were just discussed. I sympathise with the issue that you guys saw. We saw similar problems with our nameserver as well and our solution, back when I was doing this for a living, was to start migrating reverse BIND 9. However, I realise it's not necessarily a fully suitable resolution for some people. So, that said, putting on my official ICANN-IANA-person hat, we are currently working on a new registry database system that would unify management of all the zones that we are responsible for and one of the project requirements for that system is after handling of foreign reverse records or nameservers. So at least for the zones that we currently manage such as the root, which does have glue records and some of the other zones that we manage, it would be possible for such as ip6.arpa, it would be possible for ICANN to add glue records to those zones where we know what those answers should be. I'm not saying that we should do it one way or the other but I do want to bring up for community discussion that, if it were decided that that was a correct thing to do, we would be happy to do so and since, at least at that level of the tree, there's slightly less of a problem with the management and synchronisation getting out of line, it would reduce possibly two layers of delay from the excellent demonstration that you gave as to why some of these lookups sometimes extend 20, 30, 40 seconds. So that's not an incredibly large amount of substance except to say that, if the community decides this is the right solution, we would be willing to pursue that. JOAO DAMAS Thanks. We would be interested to see if that - we are a bit short on time so we should put this on the mailing list see if there are other recommendations. I would encourage the RIRs to take this to their local communities and see what the decision is. KAZUNORI FUJIWARA OK. Thank you. JOAO DAMAS The next speaker is Terry Manderson from APNIC. TERRY MANDERSON Thank you, Joao. Good afternoon, delegates. It's my pleasure to present an update on the adopted lame delegation proposal. First, I would like to briefly revisit the proposal and cover the key points so they're fresh in our minds. The proposal as adopted was that we identify the lameness of delegations by testing from at least two points on the Internet. First test evaluates the nameservers from Brisbane and, if we do not receive an answer - that is we do not receive an answer due to network timeout or something along those lines - we then repeat the test from Japan. The test, as you can imagine, simply tests that we receive an authoritative answer from the nameserver listed in the domain. Should we cover that the nameserver does not respond in the appropriate manner, we continue to test for a further 15 days before claiming that the delegation is lame. After those 15 days, we enter a 45-day period where APNIC attempts to notify the domain-holder that their domain is lame and action should be taken to rectify the issue. Once those additional 45 days pass, we then disable the delegation - that is remove it from the parent zone. The policy was implemented on the 30th of September last year, as you've already heard Joao tell us. As per policy, once we confirmed the delegation was lame, our first contact e-mails were sent out on 23rd of November. As most of you would have noticed who are any good at mathematic, this was approximately a week late according to the proposal. During the implementation of the contact process we encountered a few minor software problems in testing and chose to be cautious by delaying for due dill against making sure that any contact we made with the domain- holder is completely warranted and appropriate. Our very first lame delegation tracking in APNIC's internal systems was grated at 11:38 on the 23rd. It's good to note that the first ticket's lame delegation issues were resolved just seven days later. This is a good sign. Currently, the average time it now takes for lame delegation to be resolved is about two days. In accordance with our policy, the first nameservers that were consistently lame were undelegated on the 8th of January this year. Whilst implementing the policy, we came upon two issues that we are still investigating. The first of those relates to the lameness of IPv6. At this stage, regardless of the delegation being in ip6.int or ip6.arpa, APNIC does not check for lame delegations. There are a few valid reasons and as you'll all know, IPv6 is e merging as compared to IPv4 and there are issues of stable connectivity between the various v6 islands. Additionally, it's also been observed that some v6 islands have some - how can I say? - interesting routing behaviours and some unusual ACL lists. While this remains as an issue, any assertion that we make as to the validity of the delegation can easily be questioned. The two remaining points to cover the obvious facts that the number of delegations in v6 space is relatively low at present and any adverse effect on that in terms of the continues resolution or the latency thereof is quite small. The last issue is that, as an emerging technology, adequate research is important for the development of v6 and tests in this climate may risk the undelegation of any laboratory or research delegations. Our second issue was that there are a number of domain contacts responsible for large portions of lame delegations. As such, we're investigating the most appropriate methods and processes for contacting ad minutes that are responsible for five or more lame domains. Ideally, the last outcome we want to see is a flood of domain-related e-mail that finds its way to the administrator. Most administrators, as you know, will probably just delete it. To give you an idea of the extent of the problem, I've included a table listing the top five holders of lame domain. These are five of 185. The ad minute details have been obscured to protect their identity. It's obvious that adequate investigation into the process must happen. Looking back to the figures we presented at APNIC 18 that were generated if the policy was about to be implemented, we see that we had 168,000 nameservers to 61,000 domains. So roughly three nameservers to each domain. The top row, at this stage, an unused database flag that we have set aside for future work. Can see a number of domains are in the 45-day period, domains that, if not rectified, would go lame. Let's look at the figures from a week ago. Youll immediately notice that we have disabled undelegated sub-domains. It strikes me as interesting that the number of nameservers have decreased. Previously we had 168,000 or NS resource record. Now only 143,000 but I've added over 3,000 domains. Perhaps some people were just removing the lame or outdated servers as an easy path to rectify the issue. We really don't know. It's a pretty broad assumption. We can see that the 15-day is roughly the same. What is concerning is that this appears that many nameservers will be lame one day, not the next, lame again and so on and so forth. Such that we have a flapping effect and the lameness of the nameserver is intermittent. This basically means that we'll never progress out of the 15-day period. The numbers in the 45-day period are still quite high. This is intentional. If you recall two slides back, we have not yet progressed the greater-than-5 domains per contact into an undelegated stage. As of the 15th, we have made about 1,000 nameservers inactive. These are the ones that have not been fixed and, according to policy, should be undelegated. This is how have things gone for the proposal thus far. The graph shows that the month prior to policy implementation, up until basically last week, for all of the APNIC in-addr.arpa domains and the domains in their lame state. We can see that there are a few interesting aspects here. Most noticeable is the break in the network cable that caused lots of visibility to a portion of the Asia Pacific from our tests site. Again you can see that there and there mirroring each other. Secondly we see the good effect where one APNIC large member initiated a zone cleanup. However, overall, we observed with the slope of the total number of in-addr.arpa domains increasing, we see a very gradual decrease in the number of lames. If we think about this in ratio terms, we started the policy with about 18% lame. We have made small gains and are currently at 16%. Although this is only a 2.3% improvement, it's still an improvement. Higher returns will come with dealing with the five or more problem. During our implementation of the policy, our APNIC hostmaster team have been kind enough to share some of their problems. We've had the usual suspects of incorrect contact details or forgotten passwords. But surprisingly there are instances of APNIC members not knowing how to configure a nameserver to be authoritative. In summarising the key points of the policy impact, we have a 2.32% decrease in lame state in 3.5 months with more to come. The policy is an ongoing process. We still need to consider the time effect change on a lame state or the time two affect the change on a lame state to 60 days. We are, and probably always will be, closely monitoring the situation. Unfortunately, IPv6 support is not there so it's still a work in progress and will certainly mature as IPv6 does. Overall, the policy appears to be well received and some members have described the - it as - some members have been described as being 'appreciative' to the notice that their delegations were actually lame. Are there any questions from the floor? OLAF KOLKMAN So this presentation is DNSMON, it's a new service the RIPE NCC stands for DNS server monitoring. I'm giving this talk for my colleagues who are developing this and putting this into operation. What's the goal of the DNSMON? The goal is to monitor DNS servers and service from many places. Second goal is to do this in a neutral independent and objective fashion. This is using the RIPE NCC reputation as an independent and objective partner in the industry. We want to also present the data in a new way that is interactive and which people can explore what is happening. Why do this? Well, essentially there is a lot of measurements going on, mostly of the form of one probe somewhere on the Internet measuring one DNS server somewhere else on the Internet. Most people when they probe DNS server because they think there is a problem, there is a perceived problem will use tools like ping or any other tool that's at their disposal but mostly this goes from one location. What happens often is that people use these measurements, these measurements are used by the people themselves, they're going to call help desks and say, Your service is bad, and sometimes the press or regulators use these sort of data for policy decisions to make bad press for somebody. Better measurements, something that looks at the DNS server from multiple points on the Internet. Because then effects that are close to the measurements probe can be removed. You can see what's happening with the servers and the service by looking at it from multiple locations. It sort of averages out the outlining effects. When we do this it's by using real DNS traffic. We're not sending pings. We're sending DNS queries and we receive an answer. The measurement stations that we use are measurement stations that we have fielded already in the test traffic measurement servers. This is essentially the building block for measurement. It's a graph, days, this is weeks of data, and this is the delay for a packet going back and forth to a particular server. The delay is normalised to 250. So this would be 250 milliseconds. What we do is use these graphs from multiple locations in the Internet and we put those on top of each other and these are the multiple locations. You see we have many boxes in Europe coincidentally that's where the RIPE NCC is located. There's a box in Japan, there is New Zealand and Australia, and a couple of boxes in northern America. We hope to expand this grid to the empty areas area, empty areas here and here. (Indicates on screen) so that we have a broader field. So what I said was that we have this basic building block, this one graph, delay versus time. What we do is present that in several views. This is what we call the server view. That shows the quality of service provided by the server to all probes. So what we do is from all probes we stack these measurements on top of each other. Here is the measurement you just saw. It has been collapsed into one blue thin line. From all kinds of other positions on the Internet, all these lines could represent one server, we have about 65 servers, test servers querying. We stack these measurements on top of each other. We also colour code these measurements. If there is a normal service that if there's no more than 90% answer loss then the colour is essentially blue or light blue to make a distinction between the service, but if the query level is more than 66% then we turn the instances yellow and if it's more than 90% queries then we turn red. So here you see a little red dot and here you see some red stuff, and that means that for this particular test box, this particular measurement probe, you will find there was a small amount of time. Here is another view this shows - it might be difficult to see from the back of the room. This shows a vertical feature, there is a narrow red line in each of these graphs. That essentially means that ns.ripe didn't give answers to all of these boxes, so ns.ripe was out, it didn't work. For a period, probably a reboot or maintenance. It's very hard to see because it's such a small time scale in which it happened. This is another example. Now we get to the interactive part. This graph shows sort of the delays. We have another way of presenting that. If we only present the times that the responses were - the number of unanswered queries were worse than either 66% or 90%, to only present those cases on a different coloured background. Then you'd see that the features in which there are problems stand out much better. So what you see here is whenever something is red or yellow on the green background there is a problem for one of these locations. And here, for all the 65 locations ns.ripe.net is not visible. This is an instantaneous view on how the servers of your nameserver is perceived for all kinds of hosts around the world. You can actually zoom in and see how long this took. The time here was 10:10, here it's 10:15 so this was a two-minute outage. Probably a reboot or something. This is one view of the data. We also can present a domain view. Normally a domain has multiple servers that are located all over the Internet, so for a particular domain server, ripe.net you could have a server here, there, there. Domain view summarises the quality of service provided by all servers serving a domain. This is where all the services are located. What you see here are the name servers for a particular domain, so this is - you can not read it here but this is says ns 1, this is ns 2, 3, 4, 5, 6. This also shows when there is packet loss for on average from all these boxes from around the world those have more than 90% packet loss for this particular server's server they don't see it here but what you can see in this graph is that for this particular domain, which is in this case top level domain, the service itself has been covered. Most of the servers at this particular instance, all the other servers continue to work. So the service DNS continue to work for about an hour. If this would have been a red bar from the bottom to the top then you could've said there was no service for .nl. This is a good graph to counter the claims that a particular top level domain or another domain was not functioning at the time. You can point to the service and say there is a neutral third party that does this measurements on a regular basis and they have not seen this. That's the idea behind this. This is the third view. This is the so-called probe view. This is when you look at the results of measurements from a single location, so you go to the box in Japan and you look how all the particular servers that provide - what you see here is that box zero is atns.jp and box 22, that's something that - another nameserver. You can see these all independently. So there's different views on the data. You can brush through this and look at how things are performing from a particular point in the world. So you can trouble shoot to see if, for instance, a particular server cannot be seen from a part in the world. If people call you from the US and say, I cannot see your nameserver, you can go to this and find out if that is really the case. What we do not measure is DNS queries used for actual name resolution. What we send out are essentially measurement probe. We do not measure the total DNS service quality, or the user experience. The presentation you just had on resolving is a particular example of user experience. That is something that cannot be measured here. We cannot measure how quickly a user gets back an answer for a question they ask. It's not completely global. It has 60-plus measurements point but the bias is really through Europe. We have many boxes in Europe just because we started there. What we will not see are effects that last less than about a minute. That's the resolution, otherwise we would be arresting all these name serves and that's also not the intention. DNSMON is intended to be for network operators, LIRs, ISPs, RIPE NCC members and it's a paid service. You can get it as a paid service. I'll come back to the benefits and what you can have. TLD administrators it's for, and for the Internet community at large. That includes governments and regulators that often make claims about the quality of DNS that are not substantiated. If you are a TLD administrator or an ISP you can obtain data about quality of service, if you have poor service essentially because normally you provide DMS service to your clients. By doing so you can also improve your service improvement, you can see if you have to put in more name servers or if you have to position your name servers the different locations. You have documentation of problems and non-problems. If the press comes to you and say .nl has not been reachable for an hour on this particular date you have documentation for if there was or wasn't a problem. You have a way to demonstrate your quality of service by pointing to this neutral third party web site. It is a paying customer. If you want to do this as a TLD we're open for all TLDs. You do not have to be from the RIPE region. Also here in this region if you're a TLD operator you can come and get the service, it will cost you between 2,000 and 6,000 euros per year, that depends on the size of your operation. What do you get? It's non-exclusive. Everybody who is participating in this, but also the general public, will see the data. You can not get this service for yourself. That's impossible. Benefits - you have credible third party monitoring. You have a web site and a help desk with service level guarantees. Presence of - on the DNSMON web site. You have a link there. You can put comments in, that's still to be implemented and you get the realtime data, as it is measured you can see this data. You can influence the development. If you want to have particular features you can do a feature request and paying members like TLDs will get their request on the bop. You can also participate as a network operator, as an ISP. To do that you have to install a test box from your network for that to get DNSMON service but you can also get other things our test boxes do and that's measurement of network performance in forms of delay, loss, jitter. You get network performances from all the other boxes in the grid. It's a service you get for free. Independently monitor of critical service essentially. You can get better understanding from your customer - for customer problems. If your help desk comes to you with a question about DNS you can point to this help and help the customer and explain the problem. What do you have to do if you're an ISP or network operator? You have to buy a box. Which is a commodity, 2, 500 euros of hardware, you enter into a service contract for us for which you get maintenance of this box and access to the service which is a thousand euros per year fee. This is available for everybody. Essentially you get the same services for a TLD but you get and NTD server as well. Internet community. Everybody in the Internet can use this service. It's free, available at the dnsmon.ripe.net web site. It monitors the key infrastructure. Go there, look it up, it's nice. The raw data is available for the Internet community but only on request. I think you have to sign a non-disclosure, I'm not quite sure about that. The data that's available for the general public is delayed by two hours. So that you as a network operator who buys the service have an opportunity to fix stuff if things broke. It's free and it's a support on a best effort basis. The people who pay of course get priority. What's the time line for in? There is a public beta, you can log on to it as you probably all did a minute ago. It's been a useful service for over a year, it's been running as a beta for over a year. As of March 1 we have serviced people at the RIPE NCC, an operational service, it's operated by the people who operate all our other services as well. It's production quality. Requests for features. They are always welcome. This is where you can find more information and I would really like to invite people from this community to have a look at the service and consider acquiring such a box because we really would like to have measurements points in this area as well. It gives you large benefits, I think. That's marketing talk! Are there questions? JOAO DAMAS Questions? I do. Maybe this one is actually in Axel Pawlik. TTN boxes have been employed primarily in the RIPE NCC service for a year, but you are trying to expose this as a benefit to TLDs in general worldwide. I'm wondering if you are planning deploying this service in non TTN boxes and boxes that you install yourself. AXEL PAWLIK That's a possibility. We haven't thought much about it, but thank you. YASUHIRO MORISHITA (from JPRS) I have no question, but one comment. I have some kind of requests about the monitoring, so how about - so some kind of DNS data consistency, for example, SOA data, we want to, so external monitoring checking our DNS server's consistency. OLAF KOLKMAN You mean you would like for instance to see if all your DNS - DNS servers serve the same SOA record? YASUHIROMORISHITA Yes. So we are now continuously checking some kind of JPDNS data but we want to also check the external side of checking. JOAO DAMAS Any more questions? Time delay consideration. The fact that whenever you expose a real time meter for an Internet service to the public there is always the chance that this service will be exposed. OLAF KOLKMAN I see the monitor go (Clicks) JOAO DAMAS Then the next one says - the attack is even better. So the question is actually whether the - is there enough of a delay to sort of deter this sort of behaviour. ANDREI ROBACHEVSKY, RIPE NCC Two comments. First of all, DNS monitors is not realtime. And it cannot be used for realtime. Delay may be up to one hour of data published on the web site. I mean realtime data. Now, with regards to DNSMON service we give access to this data to DNSMON customers two hours earlier than general public gets this. This is done mainly that DNS operators can fix problems before they get questions from general public. With regards to this two hours, is it good or bad, this time was chosen from a different requirement, from the requirement to give operators time to fix the problem. If you still think that, like, two to three hours delay in data publication is too small for abusers to see results of their work and enjoy it, yeah, that's a good comment. We may consider this. OLAF KOLKMAN I wonder how much it differs if you have three hours delay or a day delay, if it shows up it shows up and they can still look at it and say, "That was me!" JOAO DAMAS I was asking the question from the point of view of a nameserver operator who is monitored by the service and if I were to join the service, if I had a choice, whether letting you do this or paying you and asking you to monitor mine would I expose myself to this kind of problem. ANDREI ROBACHEVSKY Would you Like me to comment? OLAF KOLKMAN No. Shall I do the second one? The second presentation is an update of DNSSEC. I have 10 minutes and I'll try to do it in ten minutes. This is an update on the status of DNSSEC. I'm trying not to go into details. I'm also doing this partly with my hat of DNSSEC's here. Just as something to fresh up your mind - what are we doing this for? Well, as you all know, the DNS data flow is about like this. Data goes into the DNS via registry, a registrar. Essentially through a provisioning system. At some point it will hit a zone file and the data will be published in master server. Another way to get data into the DNS is via a dynamic updates. This is how data flies through the universe. Once data hits the master server it will be communicated to slave servers to have this nice and resilient server structure. At the other side of the spectrum we have the caching forwarders. They do the questions on our behalf as clients. Our clients all have resolvers that ask the questions. If you now look at what the attack model on the DNS is - it sort of looks like this. What can you do in the DNS? You can corrupt data input. You can sit in this provisioning chain, you can hack into the zone file. You can do unauthorised updates by impersonating the client. You can impersonate the master server, you can hack into the slave servers and change the data content there. You can do cache pollution either here or here, cache impersonation. There's all kinds of places where you can do bad. Actually, a very nice presentation was given the other day by Yasuhiro Morishita showing how to do cache impersonation on a wireless LAN like this, he is able to with his laptop sweep away all your traffic very, very easy. It's somewhat harder on this side of the picture but nevertheless it's possible to corrupt data in the DNS and make people go elsewhere. Why would people do that? Why would you corrupt data? Well, primary reason - monetary value. If you can snoop away emails, redirect an email stream to your mail server by publishing a fake MX record then you could read company confidential mail. And since you pipe it back to the original target mail MTA nobody will actually notice. So there's monetary value. As soon as domain based protection against spam is in place the DNS will become a target because there is monetary value in phishing, And monetary value in spam. ENUM, there's monetary value in ENUM. I don't need to say that. There's also monetary value in attacking the ENUM stream. What does DNS provide? It provides data security, you essentially do somewhere in the provisioning chain you take your piece of data, you sign it, indicated by this. That goes into your zone file and once it's in your zone file no matter where it ends up this padlock is still on it then you can check your original data. So this is what DNSSEC provides. DNSSEC has been developed in the IETF, it took about 10 to 15 years to get it done. It's now about to be published. It's expected to pop out any moment now. Maybe even before the next IETF, otherwise shortly after. The specification. Essentially describing maybe you can do signing, how you do the serving, how the data listens to DNS and how it can be validated. There are implementations. There is a 9.3 - I don't know the current version there - 9.30. Not one? Not yet. NSD 2.1, 2.2. I forget the minor version numbers here but they all do DNSSEC. That's on the next slide. And there is an extension if you wan to develop your scripts to do scripting for system maintenance. So there's software available to start deploying a DNSSEC, there's also other authoritative service. It can be deployed today. But you will get your hands dirty. It's difficult. And there are some main improvement areas in where the IETF is working on. These are improvement areas and it might take a while to get this done. If you're looking at DNSSEC then don't wait for this to be finished. It might take a while. If you want to get your hands dirty on DNSSEC it's very good if you start now. Main improvement areas - the last mile, key management and key distribution and NSEC walk. The last mile is essentially the server in the network will do the validation for you. Once that is done how do you get the information about the validation to your application so the application can make policy decisions? There are numerous reasons why a DNSSEC validation can fail. For instance, the signature has a specific lifetime. At the end of the lifetime the signature is not valid any more. Say you did a query 2 seconds after a signature validity of 2 years. Are you going to not accept this data? I think it's better idea to accept it, but this is a policy decision that you as a user could make. It could be that you do not have all the records available for the validation. What are you going to do? Do you care about DNSSEC if you're reading a newspaper? Maybe not. Maybe you only care about DNSSEC if you're doing business with your bank. I don't know. This might be policy decisions for APP. For the application to make the policy decisions you have to communicate the result of the validation towards that application. How do you do that? That's an area we all question - where there's questions and puzzles that remain to be solved. The other problem area is essentially the key management. What will happen is a zone owner, somebody who creates a zone will sign their zones with a private key that is matched up with a public key. Public key cryptography. The public key has to go to the people who do the verification. They have the public key, they can validate a signature, they know where the data came from. In principle, you can use DNS. You put these keys in the DNS and follow a chain of trust. Down the tree you find all these keys. In the absence of such a tree you always have to configure at least one key. You might want to configure the key for the TLD, SE because they're the first ones to be signed, to be signed in September. If you want to validate their data you want to have their key. Now, what happens if the key changes, you have to get to know that. There is no technical mechanism yet to be able to see that. You have to sort of know the key policy and you have to track that as an operator because of the validation. There might be many keys living around. This is an area where work is being done. This is a graphical - say that you have os.net, which is signed and you have kids.net which is signed and you have operation here that is signed zone. To be able to do the validation you will have to get those public keys to that box. If you have many of these trusted items you get many keys in the box that you all have to track for rollovers. And there's no direct link between people who operate this and the people who do the validation. But you want to have automatic mechanisms and those are currently being thought of and looked at in IETF contracts. This is what I essentially already said. Another problem that is on the table is the so-called NSEC walk. When DNSSEC was being developed privacy was not an issue. The people who developed it always were of the opinion that if you put data in the DNS it's supposed to be published and supposed to be public. Now, at the last stages of the DNSSEC development some big TLDs from Europe where there are very strict privacy laws stepped up and said we have a privacy issue. That relates to the NSEC walk in the following way - NSEC is a record that proves the non-existence of a piece of data. The way you prove that is by building a chain of things - names that do exist in DNS. Say that A and C exist in the DNS and Z exists in the DNS. And to prove that a letter in the middle, M does not exist you build a chain from A to B saying between A and B there's nothing and from B to Z same - between B and Z there's nothing. If somebody somebodies for N they will get back a record saying between B and C there is nothing. Now, since these NSECs jump from all existing names in the tree you only have to get one and you can pull them all. Go from A to B, from B to Z then you have all these records in the tree. So you get all the zone - it's a sort of zone transfer you can do in this way. This is a problem for some big operators and that has to do with privacy constraints. It was not a requirement to solve that in the current DNS. So this is an improvement area that is currently in the IETF and people are looking at. There are a couple of possible solutions on the table, but don't expect anything to come out in the next 6 months or so. This will take some time to develop. What is important to know is that if you deploy DNSSEC today the requirement for the solution here is that the people who deploy today will not be bitten. The people who deploy DNSSEC today, server operators will not necessarily have to upgrade as soon as a new version of NSEC is being deployed. The old and the new validators will still be able to check the quota. This shouldn't be a barrier for deploying. That's the bottom line. It's also the conclusion. DNSSEC deployment can be started now. SE is far in preparing their deployment. I've heard they're going to start signing around September of this year. So I think we can safely say that by the end of the year we have the first top level domain as a signed zone. And improvements to the DNSSEC will come. DNSSEC is as a protocol more or less done, but there will be people that build on it. Things will be built upon. And those steps may take one, two, three years to be developed. If you're interested in DNSSEC, the best site to go to is www.dnssec.net. There is a dnssec-deployment.org site. We have written a DNS how you can configure a server, how you can test your set-up. If you want to read something about why you want to do DNSSEC there's a little article in the APster journal. That concludes my presentation. Four minutes to go. Are there any questions or comments? JOAO DAMAS The next speaker is George Michaelson from APNIC. GEORGE MICHAELSON This is an update on ERX activities. This is about the number of address numbers allocated or assigned before we had an RIR process. ARIN inherited maintenance responsibilities from InterNIC in 1997. There were requests from APNIC, RIPE NCC region members for a delivery of service within the region and this led to a general distribution of these resources. Notice that 110 AS transfers into our region was done as a single activity including some ranges so there's actually more discreet AS numbers than that. We began a process of transferring IPv4 addresses in January 2003 across all of the registries and final transfers for our region were done in December of 2004. So to give you some statistics, we have 16 /8s that are managed as a result of direct allocations from IANA. These are the spaces that we give out fresh addresses into the community. There were about 42 /8 s that were involved in ERX that were relevant for us. There may actually be more but they would have gone specifically to the RIPE NCC or LACNIC or some other agencies. So these are 42 that have allocations or assignments in our region. One extra /8 was only about Japan and was dealt with separately out of this process. But it's not impossible that there will be some interaction on that network in line with the way we wish to manage resources. The implications of the DNS for this is that we have direct management of two /8s, 150/8 and 163/8. There were 40 other /8s but we have to supply fragments to other RIRs. The NIRs had responsibilities that are really spread across all of these 42 /8s. About 2,300 discrete networks were moved in this transfer process and there were somewhere around 1, 200 new organisations that APNIC has had to deal with. It's hard to be exact because there are some overlaps. In some cases, an entity might have two or three different records that might have slightly different records with ARIN but are the same body. Some of these are different entities with different names, different structures. So the actual number is a bit unclear. There's been some adjustment. I'd like to point out that RIPE NCC have had a much, much bigger workload here. We were very lucky in the extent to which we had structured information but the volume was much smaller. In the main, we did these transfers across all of the RIR but 192 in a swamp, we did this separately, we had a different process. Sorry, this isn't a very readable graph but, to give you a handle on what we did - these are the list of the /8s and these are the number of networks that were transferred in that case. The one s that are in a slightly different colour, - it was only 99 or 59 network instances. The amount would have been enough to make us the majority holder. This is net 192. There were radically more instances in that block. That's why we use the different processes. OK, so what for the future? ERX NETs are subject to APNIC policy and the two that are most affected are the lame DNS and the historical resource protection. In the lame DNS, we believe we imported many lame name servers. In Terrys presentation, in the course of the sweep activity, we've had somewhat less of an outcome. We've held back on contact people but we increased the amount of lameness in the APNIC community quite markedly as a result of this activity. With historical resource protection, people are going to notice that resources that have appeared until our control, where previous they would have had a relationship with ARIN that gave them direct management, because of transference in this process, we have resumed operational responsibility. They will have to go through some activities with us. We've had some adjustment of process and have recognised the fact that people sometimes just need to get one change done so please don't interpret this as meaning that you can't fix obvious problems without going through procedural activity. There are some things that you have to do. It will be recognised there are cases where people need our assistance to do things with this resource and - so talk to us about your resource but be aware it's subject to the historical resource protection. There also has to be a methodology for transfers after ERX. What we're going to do is adopt a process already in use between the RIPE NCC and ARIN. One of the APNIC host masters, Anna, has been looking into this and I'd like to thank Leo Vegoda for his assistance documenting a procedure that looks useful to us. In a broad sense, we think this is what we're going to do. In the result of a resource needing to move into the NIR, at this stage, we would like to say that this is a case-by-case process. We think we need to speak to you to understand the particular circumstances, so I don't want to say that there's a formalism about the transfer here. It really is a reflection of differences in the NIR's DNS management and resource management just how we will make this work. But I think again the NIRs understand this activity and they know to speak to a hostmaster. We also have this outcome that we have to share zone management for the /8s where it's split across multiple registries and APNIC has been doing improvements in its DNS management processes to try and make the cycle time faster, more reliable, backed in a database and we've been in a dialogue with Shin Yamasaki from JPNIC. Again, I'd like to thank him for that activity and his thought in improving the process. And really, the only thing to say is that this project would not have been possible without Cathy Murphy At ARIN who had a huge amount of oversight and commitment to this activity over the whole period. She provided detailed transfer management, did data preparation and data checking at each stage after transfer. I'd like to thank her for that. It was an amazing, well-managed project. That's it, folks. JOAO DAMAS Anyone have any questions or comments? No? This is slightly provocative. First of all, it's nice to see closure on this process. My question is, how did this technology of transfers between registries in place, would this potentially open the door to existence of competing RIRs? Is there a need for them to be regional? Should there be more than one and you could choose who holds your reserve? GEORGE MICHAELSON I haven't been briefed to anticipate that question. LAUGHTER JOAO DAMAS OK, there is one more presentation. I realise we are now 12 minutes over the expected time to finish this SIG. The presentation is actually one I will deliver. I would like to apologise that we are over time. I will try to be brief. This is a quick overview of F-Root anycast deployment status. OK, so quickly, hierarchical anycast is a project that has been running for quite a while. The technical process is described in ISC-TN- 2003-1 which you can obtain from the ISC website. It makes a distinction between local nodes and global nodes where local nodes have a horizon, which is limited, and global ones are restricted. Most sites when we refer to anycast deployment - recently we added one here in Japan, in Osaka, hosted by - sponsored by NTT communications and it's present at two of the Internet exchanges there. If any Japanese ISPs are in the room, can they peer with those references there? Coming soon is one in Amsterdam. The intention is to small more local nodes and possibly global nodes too. We are trying to provide data to member organisations so a bit of data analysis can be carried out. So far, it's OARC. In regards to BIND, there have recently been three releases, namely in each of the three families of BIND that are still under development. BIND 9.3, which is the recommended version for everyone to run right now, if you can. BIND 9.24 which was a fix on BIND 9.2.3 but no new features will be added to that any more. And BIND 8.4.5 which was the fix release for BIND 8.4. BIND 8 is considered at end of life for us. We don't add any new features to it and we only perform critical updates if there are huge problems being deployed in the Internet. Similarly with BIND 9.2 - and BIND 9.3 is the new live tree being worked on. Just in case you are not aware of what's the deal with BIND 9.3 and why you should use it over 9.2, it includes the following new features - DNSSEC bis, both server and resolver are implemented there and available for your use. DNSSEC now you can use this. It does have extended support for IPv6. Previous versions of BIND 9 had support for IPv6 in that they could serve it but they were limited in the number of controls that the server administrator had in its hand to actually provide the service. The range of controls was much larger for v4-related information. So now, there is a parallel. They are in parallel. And so you can finetune how you provide the service in v6. There is a stricter enforcement of maximum cache size. There was were some problems with people overusing their size in BIND 9. You might end up using old memory you had in your system and it could start swapping. That's not the desired situation. So there is now a working solution for that. You can set how much memory you want your server to be using. It includes support for identification of servers as specified in an Internet draft. There seems to be a rough consensus that this is a good thing to happen so we went ahead and did it. You can control the order of rrset are returned. You can choose whether you want v6 or v4 records. You can control what goes in the initial sections. Another control that has been requested by several people is to present answers selected from different views as supported by BIND 9 depending on which TSIG key was presented for the transactions. You can automatically select information returned by the server depending on whether, for instance, the number of users are using TCP and other users are not. There is a couple of more features added that control how you operate when you use incremental updates. The case before that journal size could grow out of bounds and cause problems. You can now limit that. That gives an overview of the current state of that. Any questions? Comments? If not, then this is the end of the DNS Operations SIG at this APRICOT 2005 meeting. Thank you for being there, for coming and for your comments.