APNIC 22
APOPS - Session one Wednesday 6 September 2006
11:00 - 12:30

PHILIP SMITH: Good morning, everybody. My name is Philip Smith, I think we probably should try and make a start with this. We will wait for everybody to get settled. Yes, my name is Philip Smith. I'm one of the facilitators of this session for the rest of today. Doing something slightly different from, I guess, any previous APNIC meeting that has gone before this.

As you may or not have realised, we have had basically a birds-of-feather session that has been the APOPS meeting. APOPS is basically the Asia Pacific Operators forum. It has been a mailing list for a long, long time, probably since 1997, 1998. In about 2001 we started holding small, what I would call, birds-of-feather meetings alongside APRICOT meetings and the APRNIC, as an opportunity for service providers in the region to talk about operational issues that face them from time to time.

So APOPS has got busier and more popular. We have seen the arrival of the special interest groups. We have had the APNIC, policy SIG, database, IPv6 technical SIG and so on. What we are trying, this time around, is holding all the main operational content as part of the general session.

Before today, APOPS has been an activity that issues. Sitting on the stage with me, and I have been managing on a six-monthly basis, a fairly informal activity, a mailing list of probably 300 or 400 subscribers, low traffic, which is partly a blessing and partly a shame, because we would like to see more activity on the list from the region. But that activity means if you are not a member of the APOPS list, I'll put the details on the screen to hopefully encourage you to subscribe.

Getting back to what we are going to be doing today, we have basically contributed to APOPS as name and part of our content to the APNIC general session. Also, the APNIC special interest group chairs have participated in this activity. We have formed what is basically a program committee, the chair called for a content to the general community a few months back, soliciting for contributions. We have had a spectacular response to that.

We have some excellent presentations for you for the rest of today. Partly as a result of this, the routing and database SIGS won't be meeting at this meeting. That is one of the sadder things of that. The special interest groups, of course, exist. They will not be holding a meeting, we are going to discuss content with the special interest groups appearing within the APOPS program. That's really a bit of the background.

Before I begin the session, I would like to thank all the presenters in advance for the contributions, the special chairs, and for their assistance, and thank you to the APNIC Secretariat for the support and encouragement, as we have made this fairly major adjustment to the program. This is an experiment, if you like the format, please tell the Secretariat and let us know, as chairs. If you think it works, we will do it again. If you think it is a bad idea, we will go back to how it was before. Without further ado, we are going to have three presentations this morning between now and lunch.

One thing with the agenda we have put together, we have allowed plenty of time for questions - at least 10 minutes of question time per presentation. One thing that Randy and I experienced in the routing SIG is that we have lots of good presentations but no time for questions, so we will be deliberate to make sure there is plenty of time for questions. This is the agenda for this morning. I would like to invite the first speaker up, who is Kae Hsu from Seednet, who is talking about Internet service provision.

KAE HSU: I will introduce redundant Internet service provision. I come from Seednet. My job is to maintain the design and work of Seednet. Sometimes I support our business unit to help our customers to decide their work.

Experience from our customers' requirements. Today I am going to take the chance to share those experiences here.

Agenda - first I will show you the requirement of redundant. I classified the types of redundant into back-up, load-sharing and multihoming. I will introduce you to the challenges to service providers and solutions for consumers. Another redundant issue - MPLS and VPN. Internet is a very, very important element today in the world. Internet access is very important - access today for enterprise and business.

The tools of operation, enterprise could give Internet to decrease in cost and increase in and revenue. For consumers this is a communication tool. And for many people the Internet is the very important source of communication and entertainment. The result is customer need redundant service provisions. But, this was expensive to build redundant technology before. The reason is the leased line ISDN for back-up only. If we used it, the obligations of the circuit was low, we had to purchase some expensive network equipment. We have complex network operations. Today, we have some different types of circuit providers. FTTx, xDSL and wireless. More and more cheap and efficient network equipments appear.

We could provide enough redundant service for customers. The redundant circuit is the key of the backup. We will use the expensive backup circuit for primary use and cheap circuit for backup use.

But the fact is backup circuit only used when primary circuit is failed.

The traditional circuit backup topology. The leased line, when the primary circuit is broken, the track will take us through the ISDN. Some customers need cheaper and higher bandwidth backup solutions. Hence, we have a new circuit backup topology. We can impress the circuit to the xDSL. The use is higher than ISDL. However, we can request this line and the ISDS circuits. In some cases, the customer only uses small requirements. They can use xDSL for the primary circuit and use some wireless solution, like 3G, for backup circuits.

But some customers tell us that if they have two permanent circuits. They hope to use the two circuits at the same time. They don't want just a backup. So we have to use load-sharing. The redundant circuit is cheap, and we will use the same type of circuit for load-sharing. Sometimes two circuits with different types, but the bandwidth is possible. The sharing traffic among the circuits is the goal. Sometimes we can provide redundant PE or CE router. It is an option.

The best topology, we use two leased lines. The traffic load-sharing among the circuits. When one circuit is broken all traffic will go through the remaining circuit and sometimes a situation will happen, some party of service need to try the router. If we want to put anything of value on the router, an ISP router, we can use redundant router, too. Unfortunately, some customers think to connect their circuit to only one service provider is risky.

So they need redundant options of service providers. We introduced the multihoming. In multihoming the redundant circuit is needed. Redundancy and the redundant service provider, sorry, redundancy. There are lots of documents. I will discuss real case that our customers provide to me. An IP block for multihoming service, so our customer have to apply. There are three ways to provision multihoming service. First, BGP to exchange routing information with service providers. The customer use BGP and the routing information service. The final is the customer do not use BGP and they don't have to meet AS number. If user, if our customer have AS number, service provider, their AS number and announce to Internet. The service provider will announce, maybe for the Internet. This is a business policy.

In this case, customers have to apply a standard from RIR. Sometimes they have to maintain the BGP by themselves. Many engineers in customers' sites have fears about BGP. They tell me to maintain a BGP is very difficult and makes some pressure to them. So we have any suitable solution here?

Okay, if customer would not doing service, they can use private AS number to exchange information with ISP.

Service provider announce Internet routes. In this case, customers don't need to apply AS number. So they can save some time to deploy the multihoming service and maybe some, send some money for RIR if they are not a member of an RIR. But, the BGP issue exists.

So do we have any solution here? Okay, in this case, we don't need BGP anymore. ISP could act, customer in BGP.

Announce the customer route to Internet. The customer only want to do is maintain their routing by themselves. Service provider did not announce any routes to customers. In this situation, some customers hope, have their IP blocks increase in the real service provider. So if they have any problem in Internet access, they know who they should find. If we want to keep this use, if we have more redundant option, we can use.

Okay, challenge to service provider. We have to do something in increments for the new circuit. We need new routing architecture to control customers' routes. In old backup topology, backup route would not appear. So the primary way of router, to ASP. When a primary circuit is fail, the ISDN dial-up is change the routing status.

The ISDN line is broken. The original one is withdrawn and new customer line is announced by a backup route. But if customer used two permanent circuits, the primary and the backup line will exist in the network at the same time. So we need to differentiate between the primary and the back-up route. In this case, we can use the IBGP for backup. For some service providers, they have to reconfigure backbone routing topology. The redundant load-sharing topology, suitable routing architecture is necessary. For example, we can separate the customer into two small pieces and use different log on preference to control the traffic of the customer. At this time, the customer have to maintain it by themselves. The solution for consumers.

Only enterprises will use those redundant solution above, because we use maybe larger, expensive equipments.

Sometimes consumers would use Internet access like health and medical care or small business. So we have any solution for consumers to own their reliability Internet space? Okay, there are many multihoming gateways in the market. Our customers could purchase this multihoming gateway to get their redundant network. This equipment includes much more functions like load balance, security, VPN, quality of service, and so on.

Okay, consumers should only by the multihoming instruments and install them in his network. They can connect to different service providers. The multihoming gateway to help redundant Internet access. In this case, customers did not need any help from us. If they purchase some multihoming gateway with higher ability function, they can prevent any single value on the gateway. If customer, if consumer does not need help from us, what is our value? So we must keep to increase our quality or decrease our costs and keep our equipment.

We talk about Internet backup, but some customers use MPLS. If they want to use backup or load-sharing solution only, they could use the architecture above, but if they want to use a multihoming solution, it is difficult in Taiwan because Internet VPN is not used here.

But most of the customers use MP will recollect SVPN for service. How can we proposal suitable solutions for our customers? Okay, this is the first one. Some of our customers who use Internet for MPLS and the Internet is not provided. If this line or data is outed, they can change. But if the customer's site number tunnel, they can change. Okay, in this case, most of our customers hope to use their Internet backup for some Internet access.

So if this line is normal, the Seednet is no more, they will use the circuit for Internet access. Some customers wish to use backup only for MPLS. They apply XDL to other providers. They use a router as to control the customer routing. So, if the primary line is outage, they still can use their VPN network, no worries.

This case is a new case for us. Some of our customers use our MPLS network, VPN service. But it is hard to apply XTSL. It might be the primary requirement is not very high. So they can apply 3G service for backup. We have a customer use this test.

Okay, what is the next challenge to service providers? Anymore redundant circuit types in the future? In future, there are more circuit types, it is a challenge to us. We provide more service provision for redundant item, BGP anycast, what is that for? We have to provide any service in the future? We have to overcome any requirements. This is our market. Only when we know what is the requirement of customers exactly, just could keep customer and win trust from them, see your needs.

This is my presentation today, thank you very much.

PHILIP SMITH: Thank you very much, are there any other questions at all?

AKINORI MAEMURA: Often it is too expensive to the customer to connect, to get the service for their increase of quality. I would like to know this kind of effort is actually the result in revenue increase? In short, is that kind of redundancy product sales ware or so-so, not so much? That kind of answer?

KAE HSU: I don't get your point.

AKINORI MAEMURA: Making redundancy is fantastic for increasing quality to the customer, but sometimes the customer cannot pay a lot of money for such fantastic things. I would like to know how your customer perceived your redundancy program.

KAE HSU: I think if the customers, Internet SS is very important to us. They would wish to make some redundant part. If we could provide some much cheaper or efficient service to them, they will happy to apply our service. If we can provide much more service and redundant, or expensive or hard to maintain, maybe our customer will decide redundant is more important over the cost, more important. So if we can provide cheap redundant service for customers, they can use cheaper redundant or Internet service and they can provide more money to me.

SPEAKER FROM THE FLOOR: What is the main customer for your redundancy program?

KAE HSU: I don't know. In Taiwan, there are many, for some companies they will use this service for their business.

SPEAKER FROM THE FLOOR: Thank you very much.

PHILIP SMITH: Do we have any other questions? One here.

SPEAKER FROM THE FLOOR: I'm from China Telecom. I want to know how fast can you achieve when the network failure, I mean, if the main link have failures, how long it will take to switch to the backbone info? If you use BGP and the redundant backup, how fast to convert, to switch to the backup link? Thank you.

KAE HSU: You want to know how long?

SPEAKER FROM THE FLOOR: Yes.

KAE HSU: For example, okay, one the customer use BGP through our, okay, the customer use BGP in their network to identify if there are any problems in the Internet. Because the network is designed by Seednet, so I'm tuning the BGP to variation time. So in fact, when this line of a customer is outage, my customer tell me they don't feel any outage from this line to change to xDSL. But it must be, this 10 seconds to 20 seconds.

PHILIP SMITH: Any other questions? Can I ask one? You mentioned in your next challenge BGP anycast. Do you have thoughts about that?

KAE HSU: I think this BGP anycast Chinese only. Not suitable for a BGP test, so is ours.

PHILIP SMITH: The reason I am asking is that anycast services are used for distributing the servers. I was just wondering if there are any ideas about what you might be doing now in going forwards?

KAE HSU: The customer will use two sets and later use the IPv4 service.

PHILIP SMITH: Okay, fine. Thank you very much for your presentation. Can we have the next, please?

RANDY BUSH: Just to remind people, sign up if you have a subject that you can give a small short talk on, sign up for lightening talk for later. It is on the helpdesk.

SHENGYONG DING: Good morning. My name is Shengyong Ding and I am from China Telecom, English is not our official language, it is hard for me to make the presentation in English.

I like to make introduction about the network. This is our network overview. As you can see, there are two backbones, ChinaNet and CN2. China was built 10 years ago, CN2 is just finished. No modcast service, however, we introduced some new technology such as modcast and QoS policy. You can also see we have networks which provided services to our customers directly. Today we have about 200 major networks. One reason is because of our traditional service. Now China Telecom is trying to convert it. By February 2005, we had about 12 million customers and 4.7 million subscribers.

This is an overview of our newly built network, CN2. As you can see, we are going to carry pieces of this over the network, such as NGN Voice, 3G. This is more detailed information about CN2. It has two functions and four architectural layers. They are high-speed forwarding plane and service-providing plane.

The four layers are core layer, aggregation layer, edge layer and service layer. There are 627 routers, covering 200 cities. 202 core routers and 201 CR routers. 12 reflectors. The devices are from Alcatel, Cisco, Juniper and Huawei. There are 152 routers, about. There are over 1,800 links, 3.4T relays and 2.9 access bandwidth.

Next I would like to talking about the practice of the network for CN2. Our objectives are deliver high quality services to our customers and provide real-time network monitors and online trouble shooting. We wanted to provide centralised and accurate management and also wanted to be able to provide systematic data for network optimisation.

We have some challenges. The common things are not standard defined what functions IP networks and systems must have. Network management protocol is far away from powerful, at present, I think. The widely used SNMP is not fit for configuration management. Much information only can be communicated through Telnet interface, which is very hard to use. Also, IP technology develops very, very fast and is becoming more and more complex. So few software companies are qualified to develop professional management tools.

Besides, we have difficulties. We want to manage the network centrally and implement end-to-end fulfilment, however, the conventional management mode in China Telecom is strongly affected by regionalism. For example, the existing ChinaNet is controlled by different provincial companies through its integral network.

We need to reorganise the team and management style. Here are some principles for our solutions. First principle is centralisation. We want the system to be central to reduce costs. We want to manage the network centrally. The monitor and the trouble shooting.

Another important principle is loose coupling. We do not want another system to implement functions. Instead we want it to be implemented in a way that the deployment can be modularised. We want it to be flexible.

Based on the previous, we make the following. There are three systems. Service management subsystem, network management subsystem and process management subsystem. The service management subsystem focuses on VPN provisioning and assurance. There are more than 200 routers. The network management subsystem just focuses on network monitors and the analysis and the P routers, not including the service route.

The process management subsystem, we want it to focus on trouble shooting, to improve our operation efficiency. This is our theme for the vision for the service management subsystem. We want the system to provide automatic and end-to-end service.

From the point of our customer, they want to system to provide a professional part, such as traffic, and let them know what is happening in the network.

These are some key requirements about service management. We want to cover more than 200 routers. We also want to be able to provide auto cross and end-to-end service provisioning. We want to be able to provide basic network data and report for each customer. We want the system to be able to provide inside the VPN report and value-added service for our customer.

Our solution is we purchase Cisco solutions as a module and we asked the local developers to redevelop it to make it more friendly and customer-orientated. It results in our service provision, from the system including resource planning and allocation. The network's failure can be linked to affected customers automatically. It can be provided for each customer.

There are some remaining issues. The first one is it cannot support complex quality issues. It cannot provide inside VPN traffic analysis. This is our vision for network management subsystems. From the point of real-time monitor, we want the system to be able to provide some automatic and end-to-end analysis. Between router A and router B, we need it to report, and the module should be able to collect different data, such as traffic, network. It should be able to locate the path between router A and router B and with the data provided, it should be able to find out the congested link and following that, it should be able to find out the most suitable application. Sorry.

With Netflow, it should be able to provide automatic report about the bottleneck of the network. This is our vision for offline optimisation. We also want the system to collect all kinds of data to do analysis and find out the congested part of the network and give some suggestion to improve our network and expansion.

These are some key requirements about the network management subsystems. Firstly, we want a system to be able to manage more than 600 routers and capture the network failure in less than just one minute. It should also be able to provide an intelligent and end-to-end troubleshooting. We also want it to be able to provide accurate resource information and to provide complex and complete traffic reports. Our solution for the network management subsystems is we purchase a peak flow of the module and the design route model. We also purchased ZhongYing.

These are results our network subsystem reduced. All basic network alarms are collected and effectively processed. The link that changes in ITP can be reported in less than one minutes, thanks to the router explorer system. The quality and the results can be viewed from the system very conveniently online. The traffic is under surveillance using flow sampling technologies.

Of course, there are some problems with the modules. Firstly, the data from different modules still can't be organised well for trouble shooting and analysing. We find it very difficult for us to consolidate all the disparate subsystems at present.

The full-measure end-to-end tests haven't been deployed due to routers and system capacity limitation. The network organising system can't support configuration and analysis. This is our deployment overview. As you can see, there are dedicated server, which manages the device. We use a server to collect data. We use two routers in the network.

It enables network to monitor traffic on the network. We also employ dedicated traffic collector to collect, trap and analyse information. This is some snapshots. As you can see, we can view our topology. There are two backbones, the left is our China network and the right is the CN2. You can see it is different.

Okay, here are some lessons from the project. Management project was launched in 2003. It took three years and about four million $US to build the system. We found that managing the system construction is much more difficult and challenging than network construction, in some sense, and the comments of work companies are not so qualified to understand Telecom's requirements and technology. We think the third-party software providers must be able to provide convenient and public APIs for further integration.

We should not expect network management systems to be perfect. Instead we need to be more patient. Here are our next plans for the system. Firstly, we want to strengthen the management function for our customers. We want our customers to be able to know what is going on in their network, such as the traffic and performance. We also want to use some tools to provide modcast. We also want to introduce automatic trouble shooting. Currently our network management subsystem cannot support configuration well, we want the system to be able to provide assurance. We want to employ end-to-end testing boxes to know that in that performance of our network. We want to introduce some P2P analysis and management system.

The slide about the network augmentation.

PHILIP SMITH: Do we have any questions?

SHENGYONG DING: I hope you have no questions - my English is not good.

AKINORI MAEMURA: Your English is good enough. My question is that you have the problem with network management systems, and the last slide shows that the vendor is not as good as you expected. Do you expect to have a new vendor system developed?

SHENGYONG DING: We do not have the internal team to develop a management system. About a year ago, China Telecom developed some tools and systems to manage our network. I don't think our ISPs - we don't think China Telecom is good enough to be able to provide professional management currently. I will try to develop some programs, to build a system for us. Thank you.

GEORGE MICHAELSON: I have two questions. I'm George Michaelson. You mentioned it was a total spend-off around $4 million, as a percentage of network deployment, this was under 2%? 1%?

SHENGYONG DING: You mean our investment in the management system? Let me calculate. Yes, I think so.



SPEAKER FROM THE FLOOR: This is a low investment, it has a huge a payback. I wonder if you were looking at using IP fix when it becomes more mature as a general traffic measurement technique?

SHENGYONG DING: Are we going to apply some technology?

GEORGE MICHAELSON: IP fix. It is a neutral netflow design exercise in ITF.

SHENGYONG DING: Currently we haven't prepared for that. I don't know whether the vendors have able to provide IP fix.

SPEAKER FROM THE FLOOR: I think it is some time off yet.

SHENGYONG DING: Thank you for suggesting.

PHILIP SMITH: Any other questions?

SPEAKER FROM THE FLOOR: I have a question. Are you deploying a modcast as well?

SHENGYONG DING: Yes. I think the modcast is ready now.

SHENGYONG DING: We haven't any modcast. Our next plan is to try get a management system to manage modcast. Something like traffic engineer.

PHILIP SMITH: Okay. If we have no other questions, thank you very much. Thank you.

Our next speaker is Masaru Akai from SoftbankBB who will be talking about broadband in Japan.

MASARU AKAI: Hello. My name is Masaru Akai. Today, I will talk about Botnet and the influence that Botnet gives to broadband ISP. First, who are we?

I already talked at APRICOT this year, so please see the website if you want to know more details. In summary - we are now known as YahooBB in Japan and there are over 5 million subscribers in ADSL in Japan. We service BB Phone and BBTV and BB Mobile Point. We will change the name of Vodafone Japan to Softbank Mobile next month and we have over 15 million subscribers.

So we are a large ISP so many access lines, many PCs on our network.

What is Bot? So these are not Bots. A Bot is a software program for gathering information.

Botnet is a Bot network, machines running programs, usually referred to as worms, Trojan Horses or backdoors and under a common command and control infrastructure.

So Japanese carriers fight against malware, including Botnet. So a company's website regularly received huge DDoS attacks. They deleted web servers from DNS.

An infected PC generates large DNS queries. Telecom-ISAC Japan contacted a company to fight malwares. What is Telecom-ISAC Japan? So, Telecom-ISAC Japan is a communication company and the ISP Industry Group that is supported by the Ministry of Internal Affairs and Communications. Telecom-ISAC Japan collects, provides and analyses information. Basically it's an information sharing centre.

Telecom-ISAC Japan works with NIRT, IPA, JPCERT, etc.

Let's see the Bot. Observation record of Botnet anti-virus software - a subspecies of Bot cannot be detected, even with the latest pattern file. It is infected with the Internet user's 40-50 Internet users is infected with Bot. When a major PC connects to the Internet, it will be infected in about four minutes. -- an unmeasured PC connects to the Internet, it will be infected in about four minutes. The band width being occupied by Bot is several gbps in Japan.

Breeding record of Botnet - wild Bot is obtained. Change to livestock to MyBot. The art was trained - check action, power and ability.

Let's make Botnet.

Such an environment was constructed. We can prepare but we need a Bot source. We Googled this. Can we change the source code of the Bot? The source code is refined very much and it is beautiful. Is easy to change. The IRC's address, connection password, join password, etc. Maybe software programmer can make a Bot.

This herder's name is Foo.

This time, it is not verified. When Bot infected the PC, Bot does the infection activity to adjoin IP addresses automatically. The main body program is downloaded when succeeded in infection and it completes making Bot. When is Bot going to IP address. And it connects the IRC server. All Bot s connects to IP server. And I am now complete to build my Botnet.

To analyse my Botnet. When the source code is seen, various thing are seen. The Bot source code is divided into six things, maintenance, control, self-defence, information gathering, attack, infection activity and information gathering.

Investigation of attack function. How to send spam using Botnet. Spammer can barrage mail from the user when the message is adopted, the ISP is difficult to detect and locate.

How to attack DDoS using Botnet. It's so easy.

Investigation of Botnet operation. IRC server is important infrastructure for herder and Botnet. To down Botnet, we down the IRC server. Bot automatically connects to back-up IRC server. The recovery of my Botnet is completed.

The form of use of the IRC server - herder gives two or more server. Channel is using control for doing spam or DDoS. Another channel is using management for software operators of Bot, etc.

Conclusion of this report. Botnet is a great system. We should be serious, because the cracker is serious. Botnet makes money.

There is no specific vaccine. It is needed to correct vulnerability in the Internet system. And we should try to detect Bots-infected PC, the C and C servers and the download servers.

The situation of BB Technology. I cannot talk of many things from the viewpoint of the information protection. Please don't go out the door. Please stop running.

Yahoo BB's DNS gets lots of queries.

[at the request of the presenter, certain sections of this transcript have been removed]

Are you ready? So hereafter, broadband network will advance in most countries. At that time, can you defend the users from Botnet and other malware? Keep secure routers, servers, switches, etc. Source address validation - ACL or uRPF. Mr Matsuzaki will explain source address spoofing in the last session of today's APOPS and recommend creating a security community in your country. It will help.

Questions?

PHILIP SMITH: So do we have any questions?

KAZU YAMAMOTO: Kazu Yamamoto, IIJ. Can you go back to the slide with the DNS graph?

I don't understand why a record was described as a spammer or virus-infected. Would that be easy to look up?

MASARU AKAI: Spammer or virus-infected is not for A. It's for MX.

KAZU YAMAMOTO: Oh, MX. From the mail populated point of view, many spammers using Yahoo! BB routers and using all kinds of destinations and that ISP should try to return unnecessary error message to Yahoo! BB. So of course Bot can look up MX of Yahoo! BB but I guess other ISP also look up MX. Right?

MASARU AKAI: I don't decide the source address this time.

KAZU YAMAMOTO: OK. Just a comment.

MASARU AKAI: Thank you.

PHILIP SMITH: Any other questions?

OK, can I ask one? Do you have advice for the community at large about how people go and set up security - I suppose groups within their own countries? Because whenever I try and push this and promote this, people say, "Yeah, but it's not a problem for me. I'll worry about it when it becomes a real issue." And that's usually when it's too late.

RANDY BUSH: Remember Geoff's curve? We're in denial.

MASARU AKAI: So... Japanese community is also late so...

PHILIP SMITH: Would it be worth the Japanese community - say taking Randy's suggestion, taking Geoff's curve and applying it to the security issue and going around the region in forums like this and just say, "Look, this is what's going to hit you. This has hit us. We are here on the curve. Please don't make the same issues that we had."

RANDY BUSH: I have two things. One is, um, the Japanese community might tell how they are dealing with this. This describes the problem. You might say what solution you're using so that other people may learn from it. But I have another question of the audience, which is how many people out here are involved, either in their organisation or themselves in security management solutions, etc, in the Botnet space? I know there are people there who are. Mars, why don't you start by raising your hand?

I think there are people who are shy. I think many of us are actually involved in trying to do this. I'm aware of the ones in Japan, but there must be other countries, the Koreans, etc, the Taiwanese, involved in Bot-chasing. There are lessons to be learned.

This shows the problem, right, and the different techniques the Botnet Creators are using. What we do not talk about is what the Botnet defenders are using.

TOMOYA YOSHIDA: Hello. My name is Tomoya Yoshida. In Japan, we share the Botnet information, one way of how to share is - one is using the mailing list. The mailing list is not open. It's confidential. So we share information which is important for ISPs. So we take very carefully that information. In Japan, the biggest ISP, big ISP faces those kind of situations so we have many Bot from the clients who already have a Botnet. So in Japan, we kill each Bot that opens the IRC port. So we kill each by each. So in Japan, many ISP s are considered to improve those situations in the - we have a simple information-sharing system, but we try to include those kinds of issues. I think, you know, in other countries, we have the same situation so, if you have any ideas, please let me know. Thank you.



NEW SPEAKER: I have a question about the speaker. Have you estimated how much the traffic Botnet in your normal traffic and have you some percentage data you can show us? Thank you.

My question is have you estimated how much the traffic generated by the Botnet in your normal traffic every day and have you some data you can show us about the damage it does to you?

RANDY BUSH: What percentage of traffic is Bot traffic?

MASARU AKAI: I have no data this time.

NEW SPEAKER: Thank you.

RANDY BUSH: Let me make a guess. Most Botnet traffic is related to DoS and spam. The spam is going to show up as e-mail traffic and as a percentage of measured traffic, it's fairly low compared to point-to-point protocols. DoS attacks, on the other hand, are big spikes and when that happens, they're a large percentage of the traffic. Otherwise not. But those are guesses.

KAZU YAMAMOTO: If my understanding is correct, a man there will make a presentation on the source information, right?

SHANKA CHAUDHURI: Good afternoon. My name is Shanka Chaufhuri. I'm from Reliance Infocomm in India. I'm from an ISP so my question is do you receive a lot of e-mails from bodies stating that our customer is spamming and they are not the mail Bots. So what I'd like to know is do the ISPs study in the situation what kind of - we have a search process in which we mail to our customers and correspondingly take effective action if they are held responsible. But then again, how much are the ISPs responsible in this? Do we have any legal responsibilities as such? Are we to be held responsible if the customer keeps spamming? Where do we finally end up in the Internet connection itself.

PHILIP SMITH: Can I just quote what Randy said to me just now? Hot potato.

RANDY BUSH: That's American idiom for "we don't want to go there". First of all, how can we speak of legal responsibility? We don't know. Secondly, jurisdiction - different in different countries. Thirdly, in some countries' legal cultures, discussing that one may be has responsibility gives one the responsibility.

MASARU AKAI: In Japan, the government propose some legal responsibilities.

PHILIP SMITH: OK. Final question, Gaurab.

GAURAB RAJ UPADHAYA: Thank you, Philip. I'm Gaurab. Actually, I was coming back to the question about how much traffic does your Botnet use. A lot of time when you say Botnet, there are two aspects to traffic - how much traffic is the Botnet using and how much traffic it is generating as a result of some other using the Botnet CNC to send spam or create DDoS attacks. And as I understand, when you see a DDoS, you don't see Botnet CNC. You just see the accident but you don't see the cause. So, I mean, I would be interested to see the cost and how much is DoS attack and how much of spam is really coming from the Botnet CNC or Bots spread out to broadband users. That would be more interesting to see and how much spam is coming from ISPs and open relays. Thank you.

RANDY BUSH: Can I use traffic analysis to see Botnets?

KAZU YAMAMOTO: Can you show the webpage of www.spamhaus.org and click on top ten. If you want statistics, please go to spamhaus.org.

If people here don't know Botnet Well, please refer to this webpage and please understand your country is generating many, many spam messages to other countries, OK? That is not other people's problem. This is our problem.

PHILIP SMITH: OK. I think we can probably finish the discussion there. Certainly you've generated quite a lot of interesting discussion after your presentation so I'd like to thank you very much for that. Thank you.

APPLAUSE

OK, so that brings us to the end of the morning session. We're finished just over time, so five minutes late but then we started five minutes late.

This afternoon, we've got quite a busy 90 minutes starting at two o'clock, so I'd like you to be back here on time if you would please.

Just a reminder, as Randy mentioned earlier on, if you've got any wonderful ideas for quite - five-minute presentation or whatever, please let the APNIC Secretariat know because we're just about to close the doors for the lightning talks section starting at six o'clock this evening. So if you've got any wonderful ideas for any sort of hot topics of the day, please let the Secretariat staff know or let any of us up here know and we'll put you on the agenda. You don't need any slides, don't need any wonderful presentation, as long as you don't mind standing up here at the microphone and saying, "Hey, what do you think about this?" that's all it needs.

With that it is time for lunch so we'll see you at two o'clock.

Thank you.

APNIC 22
APOPS - Session Two
Wednesday 6 September 2006
1400-1530


PHILIP SMITH: Welcome to the second session of APOPS. So this afternoon, the first part we're going to look at some of the Internet operational things that are happening around the region. We've got four presentations.

The first one is going to be by Sumon Ahmed Sabir from BDCOM. And then we have Amante Alvaran from APNIC who'll be talking about the industry in the Philippines. Gaurab Upadhaya will be looking at AS-paths and the research work he's been doing with colleagues. And then Merike will finish off having a look at IPv6, as in what works and what doesn't.

Now, this APNIC meeting has had quite a lot of experimentation. We're going to do another experiment. One of the things that we're trying to at least encourage people to do is to consider remote participation, or at least participants who cannot come to APNIC meeting, we're still encouraging them to submit presentations but in I suppose remote format.

So Sumon cannot come and be with us here today. He's in Bangladesh. He can't make it. So he's actually recorded his presentation and sent us the video instead. He's online at the moment on the Jabber chat so is available to take questions. And Gaurab will relay the answers that he provides to the rest of the audience here. So this is a bit of an experiment. It's the first time we've ever tried this. We're going to try and see if it works OK.

If this works fine, we'd like to encourage people to consider this as a future way of participating in the APNIC meeting. So let me go and fiddle with my laptop and we'll get the video started.

SUMON AHMED SABIR: Hello there, everybody. Today, my discussion topic is impact of SEAMEWE-4 submarine cable on Bangladesh. Recently, Bangladesh got the SEAMEWE-4 submarine cable and ultimately it has a lot of impact on the Internet business and Internet services in Bangladesh.

Bangladesh got its first Internet connectivity in May 1996 through a 64 kbps VSAT circuit. Since then, there is a mushroom growth of Internet service and the number of ISPs increased a lot, but unfortunately the number of Internet users is quite insignificant compared to the country's population.

In 2006, the number of ISPs grows to more than 150, but mostly concentrated in Dhaka city. And approximate bandwidth consumption through VSAT is around 400 mbps.

This is the diagram of SEAMEWE-4 cable system. It is travelling from Marseille, France, to Singapore and touching Thailand, Bangladesh, Sri Lanka, India, Pakistan, Indonesia and some countries in the Middle East and Europe.

SEAMEWE-4 cable system landed in Bangladesh in October 2006 but unfortunately it took another six months to reach the capital city of Dhaka. And we got our first Internet space through SEAMEWE-4 submarine cable in May 20, 2006. And at present, two STM-1 links are up, one with MCI in France and another with Singtel in Singapore.

Since this submarine cable has started, there is a growth and demand for the bandwidth. It has grown a lot and as a result another two STM-1 and STM-4 links are in the pipeline and we're expecting to both get those STM-1 and STM-4 within this year.

Since this submarine cable has started, we have experienced significant improvement in the performance, especially in voice and video conferencing. Ping time was 520 ms with satellite but now it has come down to 180 ms to 500 ms around the globe. And people started playing online games, watching streaming movies, so usage has also changed a bit and demand for high bandwidth increased a lot.

But it was not very easy to get the services and get these experiences. The major transit provider BTTB was not quite ready to provide this connectivity. It doesn't have the skilled man power to handle a transit point and there is no infrastructure in the country to connect the ISPs with the submarine cable. There are no good routers in BTTB to connect with the backbone. There are no good routers or switches to connect the ISPs and so far, there is no redundancy for any infrastructure available. To resolve these issues, the ISPs extended their cooperation.

ISPs have laid their own fibre through the city to reach the BTTB landing station. They helped out BTTB and other ISPs to configure their routers. They assisted the ISPs to get IP address and AS numbers from APNIC. And finally, most of the ISPs are experiencing Internet services through the submarine cables.

The picture you're seeing now, people are splicing fibre in the streets of Dhaka, turning on the headlights of their vehicles in the middle of the night. And this is become a common scene. Some new ISPs are coming up for the submarine cable. People are climbing electricity poles, hanging the cables, fibre optic cables, from the electric poles. There are no underground cables in Dhaka city and it's not possible to lay cable underground within a very short time. And finally now we're in the building in NOC. This picture is taken on 19 May, just after the STM-1 link up with the MCI and two of the ISPs are already connected to the SEAMEWE-4 cable backbone and we're preparing for the connecting the rest of the ISPs.

But it was not quite easy for all the ISPs. It's another link, so most of the ISPs become multihomed for the first time. And most of them doesn't have their own IP addresses, doesn't have AS numbers. And it's not easy for BTTB to assign the ISPs. As a result, APNIC experienced record growth in IP and AS number allocation for Bangladesh.

For the last two years, there was significant training on BGP routing and BGP multihoming, as two years back, BDIX was set up and SANOG V was in January 2005. But still most of the ISPs had a lot of problems in configuring their routers and some of them misconfigured their routers and caused downtime in several instances.

For example, one ISP became a transit point for most of the ISPs. It announced all the prefixes to satellite APNIC and their satellite service provider for the ISP also announced this to the Internet. Eventually, most of the ISPs were out of the Internet for about three hours.

Some ISPs got allocation from APNIC for a /21. They announced a /20. As a result, part of the Internet became invisible for fixing these issues.

Over a period of 10 or 15 days, most of the ISPs started to see packet loss, some experienced as 5%, some experienced more, up to 60% some time. And finally we found this is because some dupe lex mismatch in the router interfaces experienced high CPU load and mostly due to wrong router configuration and sometimes for the overloaded routers.

Even the main backbone router with MCI in BTTB also became overloaded and had a 98% CPU load. And to fix this, we changed the configuration and fixed the problem.

So now we've discussed all the good things of the submarine cable but after getting it, we've got some negative impacts as well.

ISPs experienced more downtime in the last three months than in the last ten years even. There are 500 kilometres of fibres overland from the capital to the landing station in Cox's Bazar. The cable was cut three times in the last three months and each time it takes around 5 to 10 hours to repair the cable. And within this three months, the SEAMEWE-4 cable was cut near the Suez Canal and it took five days to repair.

After the start of the SEAMEWE-4 cable, there was an impact on the BDIX traffic too. There were several ISPs that were connected to the SEAMEWE-4 cable came - 17 ISPs. But today I found only 12 were appearing in the access point and one month before, it was only nine. Most of the ISPs thought that, "We have got already transit point in BTTB, so BDIX is not really necessary." And some ISPs may have had trouble reaching the router, so preferred to shun the BDIX connection. But the good thing is that the ISPs are realising it now and they are coming back. And though there is a number of ISPs using BDIX, the traffic is gradually increasing and as the bandwidth is increasing, the traffic in BDIX is also increasing significantly.

As well as the technical and service delivery impacts, there are some business impacts of SEAMEWE-4 cable as well. Bandwidth cost for the ISPs reduced significantly. As a result, we're expecting that a rapid growth of broadband users and already we have observed this trend.

And some ISPs are having costly satellite backup because SEAMEWE-4 cables are still not stable and we are observing downtime in it quite frequently. But some ISPs are having delivery from SEAMEWE-4 cable only. So bandwidth price is varying significantly among ISPs. As a result, user migration becomes a major concern for the ISPs.

So far, there are 34 ISPs are now appearing with BTTB and getting the bandwidth through SEAMEWE-4 submarine cable. And both STM-1 links are almost saturated, so BTTB is getting another STM-1 within quite a short time and within this year, two STM-4 links will be up. And another 10 to 15 ISPs are in the pipeline, waiting for the new STM-1 links.

The ISP Association has taken some initiatives to train up the technical persons from the ISPs in BGP and routing and BGP multihoming. We're organising seminars to alert the ISPs about the benefits and importance of the Internet Exchange point. BTTB is procuring new hard wires to consider the future demand. But there is still a lot of things to do.

And we are learning from our experiences, sharing information with other ISPs' people but still a lot of things to do. And we are expecting to have a robust and better Internet soon.

Thank you.

PHILIP SMITH: OK. Thanks very much to Sumon. Do you have any questions for him at all? I believe Sumon is actually watching on the live video feed at the moment, so he'll be able to take questions from you.

Nobody?

OK. I've got one question. What was the reason for ISPs abandoning the BDIX in favour of just moving over to the BTTB transit point?

I mean I know they said, "Oh, well, it's not necessary any more," but why did they think that? What was the reasoning? Because basically the transit through BDIX is free whereas BTTB charge money. So I don't understand that issue.

GAURAB RAJ UPADHAYA: I think Sumon said that a lot of small ISPs only had a few routers with them and then, you know, when they realised that they need today connect to BTTB and get a new router, they just moved the router from the BDIX to BTTB, but they are slowly coming back. And Sumon is saying something... (pause) Sumon says the same thing.

PHILIP SMITH: OK. Thank you very much. No questions at all?

If not, thanks very much Sumon. I hope all the hard work putting the video and remote participation together was worth it. Thanks.

OK. Next up, we have Amante from APNIC, who will be talking about the local peering situation in the Philippines and the PH network operations group activities.

AMANTE ALVARAN: Good afternoon, everyone. My name is Amante from APNIC. I'm going to do a presentation about the current situation in the Philippines with regards to local peering and some of the work we are doing.

So the current situation now in Philippines is that there is no common exchange point. There is no common exchange point. All exchange points, all IXs or Internet Exchanges, are owned by the telcos. So we currently have four telco companies in the Philippines who, each of them, have their own Internet Exchange and they don't really peer pretty well with each other. They have some political issues within them, like bandwidth utilisation, connectivity issues, all those things. And all those IXs mostly cater for their own customers only. So those ISPs, or those organisations which are connected to the IX, are the same customers which they have into their ISP service. OK, so there is nothing much to do in the IX business at all. And because of the issues or because of political issues with different IX providers, because they are owned by different telcos, most of the local traffic goes international.

One case, for example, is that you have a customer here - you are an ISP which has a customer that is - that is using one of the telco's bandwidth and you want to access your colleague, for example, which is using a different is service provider, you have to go through the international link just to reach that particular friend of yours. So that is what's happening now in Philippines. We have IX but it's not working as an IX, OK?

Limitation on bandwidth - there is one Internet Exchange provider who is actually charging the bandwidth at the local exchange more than the price of bandwidth for the Internet service. OK. One example is that anyone who connects to the exchange, it will cost you about US$800 while an E-1 connection to an ISP service will cost you US$200 to US$300, so you will subscribe to the Internet service rather than the Internet Exchange because of price issues. And because of those political issues as well there is no available statistics so you wouldn't know what is the local traffic within the country. So that's the current issues.

And these are the IX providers in the Philippines. So we have PLDT, which is virtually a monopoly. That's the Philippines Long Distant Telecom. We also have an IX called Globe Telecom, which is now Innove Communications. Most of the customers of those two IX are their own existing ISP customers. We also have MIX. The Bayantel network, which is the National Gaming Internet Exchange, and another exchange which is owned by PLDT as well, ePLDT, which caters for content provider and gaming provider. So those are the current providers at the moment.

Since we don't have a better connectivity between those IX, as you can see, we have this problem of localising the traffic because all traffic goes international before it reaches the other service provider that is connected to the other telcos. Our goal is to have a common exchange point. This is what we are trying to do here. So that would eliminate bandwidth connectivity between these telcos since we have these bandwidth issues with the telcos, One telco could not even upgrade their bandwidth because of price issues, because of connectivity issues - no-one wants to go ahead with a bandwidth upgrade. Then of course we want to provide better statistics, OK? So we're trying to local identities traffic between those IXs by setting up a common exchange point, localise traffic between ISPs and also content providers.

This is our solution - set up a common peering point to provide local traffic and we're kind of thinking of calling it PHv46X, which means Philippines version four and six exchange. So that will address all of those issues, which I have said, and will let the local traffic be local in that sense. So from its birth, it's going to be supporting version 4 and version 6. And because of this being a common exchange, we will be able to at least provide more access to local contents, like the government networks and some gaming providers as well.

OK, PHv46X is an initiative of PHNOG, which is Philippines Network Operators Group. It was a recently formed organisation. At the moment, we have about 20 organisations as members of PHNOG with 30 individuals registered. And it's also an initiative from the Advanced Science Technology Institute in the Philippines, which is under the Department of Science and Technology. It's a government agency.

DOST comes into the picture because it will handle the operations because they have the infrastructure. DOST has a direct connection to Japan for the IPv6 network and they have all the facilities and the resources to run the exchange and they are, of course, the common organisations. There's no business interest in that sense.

The services that we're going to be providing for the v4 PHv46X are those services which are on the board. We're going to provide v4 and v6 peering. At the moment, they have a v6 tunnel broker but it's not open to anyone o so we're going to open up a tunnel broker once we set up the exchange. They have routeviews as well as available but it's only showing them the route information's with regards to their existing network at the moment. And also we're looking into hosting a route server, but we have to talk about APNIC about it, how we can host a route server inside the exchange and for the non-ISP customers we're looking into providing them a service where they can also get a AS number for the time being that they are connected to the exchange.

OK, other services we have is - this is a future plan where we'll set up a route server and to support the services that we're going that we're going to be providing, there will be training for sure. Training for the IPv6 protocol, exchange services orientation for all the members of the exchange and some collaborative training with PHNOG and APNIC. OK, those are the trainings that we are looking at at the moment.

OK, so that is PHNOG as I mentioned before. We have about 30 members from 20 different organisations. So if you want to know more about PHNOG that's our website - www.phnog.org. And, yes, so those are one of the activities - skills enhancement of members and non-members, so we're thinking like the other NOGs in different countries where we provide training to the members.

So that is the DOST - Advance Science Technology Institute and Department of Science and Technology in the Philippines. So they are a government organisation so they'll be the one maintaining the exchange. What they do is they do more on research and development for the ICT and, yeah, so that's more about what the DOST is doing.

OK, so that's the current technology thing which is happening now in the Philippines. Is there any questions?

PHILIP SMITH: Anybody have any questions for Amante? There's a microphone in the middle and there's a microphone walking around at the back of the room as well.

AMANTE ALVARAN: There is one thing which I forgot to mention as well - I would like to take this opportunity to thank Cisco and PCH and Consulintel for their support. Cisco and PCH run the switch. So thank you very much for that.

(Pause)

PHILIP SMITH: Lunch must have been too good. Everybody is sitting in silence. One question.

SPEAKER FROM THE FLOOR: You mentioned the political reasons that the carriers don't want to interconnect with each other but now the initiative to have a neutral IX - what are the responses from those carriers so far?

AMANTE ALVARAN: So far, we have got about three of those telcos, they have initiated their support. They will support the exchange project. But we still haven't got the other one yet. You know which one is that, which I mentioned before. So yes there is a good response with this project and also the ICT or the Information Commission Technology in the Philippines is also pushing for this one.

PHILIP SMITH: Is it the members of the NOG who are actually keen for the IX to happen? Usually you need a small critical mass of ISPs who want this neutral exchange and then everybody else will join afterwards.

AMANTE ALVARAN: Yes. That's a good question. All the NOG's members are very keen to have this.

PHILIP SMITH: OK, if there are no more questions, thank you very much, Amante.

AMANTE ALVARAN: Thank you.

APPLAUSE

PHILIP SMITH: So next up we have Gaurab, who will be presenting some research work that he and colleagues have been doing, having a look at AS-paths amongst lots of other things.

GAURAB RAJ UPADHAYA: Thank you, Philip. I've got plenty of time, so I can speak slowly.

Anyway, I am going to speak about the work I've been doing with a colleague of mine on the AS-path analysis. For those of you who were at the last APNIC meeting, I mean the one before Perth, in Hanoi, we did some analysis on similar lines but at that point in time we were just starting on this work and now we have some more interesting stuff to share with you. So, it's mainly analysis of the Internet routing tables as seen from many different places - routeviews, our own route collectors in various locations, feed that Philip gives us, feed that other people in Europe give us and I'm still trying to find a feed in South America. So this is a combination of analysis of a lot of routing table data and we are especially looking at AS-paths because they have certain properties. I'd especially like to acknowledge Vijay Kumar Adhikari whom I worked with on this work.

The reason we started looking at AS-path and tier-1 analysis was that we looked at the AS-path and said, "Hey, there is some amount of publicly available data that can indicate a relationship between different ASNs." So before that, there was only one thing that you could really look at and say, "OK, we can do something with this here."

And then, when you looked at AS-paths and operators, then, you know, what do you want to find out? We said, "Let's go and find out tier-1s and people claim they do not buy or sell from anyone." That's how we defined tier-1. They always tell other people that we are tier-1 because we don't buy transit from anyone. The only thing we do is we sell and in extreme cases we peer with fellow tier-1s.

So Autonomous Systems which do not receive transit may reach other ASes by selling transit to them or by peering with them, right.

So this is what an AS-path looks like most of the time. In most AS-paths, there is one centre ASN or, in this example here we've got 1239 as the centre AS and we all know that Sprint is a so-called tier-1, right? And we said, "OK, this is what it looks like. Dupont or 7823 buys transit from Sprint and then Sprint sells to SBC - it's now bought by AT&T but at this time it was valid - and Sprint sells to SBC and SBC sells to Fry's and this is what it looks like. Or it could look something like this - there are two big provider, say for example, AT&T Verio, 3914 and Sprint, 1239 and they of course peer with each other. And they sell to us, 3856 and Sprint sells on the other side. An AS-path can take one of these two forms. You also assume there is always one or two ASes at the top.

So what is the proposition - our proposition is that if somebody claims that they are tier-1 provider by their earlier definition, there cannot be more than two-tier-1s in a single AS-path. That's a perfectly valid assumption. If somebody else is there claiming they're a tier-1 and there are three of those in a single AS-path, well then there is something wrong. And either somebody is buying or somebody is selling transit. So that is our proposition.

So what we did to test this proposition - we couldn't just run a test on the entire routing table. So we took Philip's CIDR report that comes out every week and added our own step and we found ten ASNs which had the most number of routes attributed to them. So we looked at our own internal peering with a couple of these guys here. We looked at who sends us the most amount of routes and presumed those are tier-1 and then the algorithm based on those numbers.

After running it through many of these - and here I've got ten of those but we had a lot more in our list to start with, so we ran through it and after looking at it for a while, we realised that these guys are probably, you know, fit the proposition quite well or they qualify being the so-called tier-1s as per our algorithm and as per our definition of what a tier-1 is earlier and it's not a big deal because all the tier-1 guys are accepted in the NANOG community and other places where I've done similar presentations.

And there was another case with ATDN and we kept on doing this - add ago candidate. Somebody says, "Hey, we are also a big tier-1." And we got a lot more problems when we tried to put them in the list. We did that and a bunch of those analyses are already there in last year's APNIC meeting presentation. We were going to have a look at specifics there. So we worked on this for a while and then we settled down on those ten big ASNs plus ATDN but ATDN has some kind of special relationship with other providers in Asia so, you know, it's still something we need to do there.

So when we started testing these propositions, we found a lot of anomalies and what I term anomalies is, well, we know that 7018 and Verio are big providers. Even if they don't - if there is something else in between these two guys, then there is a problem, right? Now if there is someone between Verio and NTT, like Sprint or someone, there could be some explanation there but you can't expect a very small provider in South America, some really small broadband provider or residential Internet provider to be providing transit between Verio and UUNET, right? So when we did this, we found a lot of these things happening, right? And we've got an example there with that particular question. I think this has gone away now but it was true up to a couple of weeks ago, where we saw AS 6221 which is Cybersites, which is something in New York, which was, you know, being seen between Global Crossing and Level 3 and, you know, that is something, you know, most people working in this field would realise there is some kind of problem there.

You really don't expect someone else to show up between Level 3 and Global Crossing, unless it is UUNET or Sprint, right? So this is what we found. We'd been running this research for a while and instead of trying to list all the anomalies we find, we categorise anomalies and we have found that these are quite regular. An example of this is I've got data from June, when I was last running the data. So the example I showed earlier where three of these tier-1s showed up on our list, you know, that is the occurrence. It was around 160 to 170 most of the time. And that is true even now. I was looking at last week's data and it's still around 160 and 170 and in many cases, the prefixes are remaining constant and the ASes in question are always the same ASes because they get the same problem. They never go away.

In some cases we see a problem and then after a week that ASN and that prefix in question has gone away but something else has popped up, right? So that keeps on happening.

And this is what we did last year or six months ago and then we started looking deeper into AS-paths and we found more anomalies and we categorised them into four boxes - inconsistent ASNs, non-contiguous repeats, private ASNs and unallocated ASNs.

So with the inconsistent ASN - that's what is on this slide. These are prefixes announced by more than one provider. We give an example of a prefix announced by two ASNs and there is announced by four ASNs and so on. It is interesting to see 17228, 17229 and 17233 are different ASNs but they all belong to related organisations somewhere so things like that happen.

So this is what we've seen. You know, who announces the highest number of inconsistent prefixes. AS 4808 followed by AS 29257 and there are some ASNs who are really doing it quite a lot.

In a few cases, I actually knew people who ran these ASNs or I met them in different meetings and I asked what is wrong with your inconsistent prefixing? They said they are transitioning ASNs and that's happening with ARNET right now, the Australian research network and they're now seeing a pretty large volume of prefixes from two different ASNs so some of these might be valid reasons but you still see the inconsistent AS there.

This is the number of inconsistent prefixes we saw that week. Philip tells me that he saw half of that. I'm not so sure. We ran this test again after he told me that and still got that. Maybe your views and our views of the Internet might be different.

So another thing that is quite interesting when you look at AS past and I don't know how many people have looked at it so closely. Can you see that from the back, with the colours of the slides? Axel? Yes?

So these are some examples. You see there someone is doing AS-path with 12163 and there is suddenly a 12162 in the middle of nowhere. So this is what we term as non-contiguous repeats. And there is another case here where you see 7018, 701 8 which happens to be AT&T, one of those and then suddenly you see this private, you know, more than unallocated ASNs in the middle, 65000 and 65001 and there are more here, 21548 and then there is another AT&T there. You can explain each of those in your own way.

I don't really see what the problem is there and I think we got them to fix this. We sent them an e-mail saying, "Hey, what's this?" and they said, "Oh, thank you for pointing this out. We actually made an error." So they probably fixed this.

Now this is very consistent. If you look at the data, this is not a single occurrence. There are at least 40 or 50 of these occurring. And we think that somewhere between AT&T and SBC and they're trying to merge the network together, they started leaking their addresses out to the public somewhere. That's my, you know, random guess, right? But it could be anything.

So there are more occurrences like this. And this one - well, it's very difficult to say... it looks like, you know, 2154 is doing AS-path poisoning so that, you know, they don't get any prefixes from that particular AS. This one is the most difficult to categorise. Or it could be some form of leak because this is trying to do AS-path. It's something you really can't point out by just looking at AS-path. So we're saying, you know, all these three different categories where something is really an error that people make by typing, something is, you know, something - I'm going to look at this for another few months and if it's still there, you know, maybe ask somebody at SBC or AT&T to say, "Hey, you guys are doing something that doesn't make accepts." Right now they're in transition. This may be deliberate, may not be deliberate, who knows?

This number is also pretty consistent. Within a week we're seeing 2800 to 2900 AS-path problems, so, well, in is a pretty high number from what I can tell, this is not a few numbers. It's like out of 190,000 route or say 1 85,000 routes during the first week of June, this is more like 0.5%, yes? Around 0.5% of the total number of routes.

So still not significant but if you look at the absolute number - 2,800 AS-paths with this problem is, I think, is an indication of something. I don't know what.

So which are the ASes most with this problem? AS 9498 and 4802 like to do this quite often.

OK, what I was trying to show you was, if you see AS 4808 and - I thought there was a correlation but there isn't. Sorry.

These are the ASes that are most involved in this thing and where we have seen the most non-contiguous with these ASNs. Some of these might be incidental and some of these might be, you know, deliberate. You can't really say.

We also see quite a few private AS number leaks. There is 65000 being leaked in this case and this is an example from earlier. This is not really a private ASN. I think it's still in the unallocated ASN range rather than private. And then there is another case where another ASN is being leaked out and it's very difficult to categorise how that is happening.

Private AS number leak - there are a few... again, there are some ASNs which are particular there - 34383 and 24141 - these ASNs are most associated with this particular leak that we saw and again we are seeing more - anywhere from 400 private AS number leaks in the table and I think that is consistent with what other people are seeing as well.

Using and leaking unallocated ASN - this is another one. We're just seeing one and again Philip says that there are more. So we need to look at it again.

So the differences between the earlier one and this one is that, in this case, people are actually using it and leaking it. In the earlier case, it was somewhere in the middle of the AS-path, it was not the original AS. So right now we are seeing one - or at least in the first week of June - we are seeing one origin AS which is private or unallocated. Interestingly, another point of routeviews, we see a lot of IGP there leaked by providers. I don't know what to make of it.

Now looking further into the AS-path, right, as I said, other people buy, other people sell, or they peer with each other that. Is the standard Internet world.

So if you look at the AS-path and if it doesn't make sense, you can't say, "OK, this is buying and this AS is selling." You know, you'll find AS - or we have found AS-paths where it looks like both the ASNs are buying and selling to each other and that's what we term the X relationship.

So contrary to our basic presumptions about ISP business works, we see quite a few X relationships. Like this one here. Again, I've pointed out two examples.

See these two ASNs - 9498 and 9730 - and if you look at this, this guy's selling transit to this guy and so on. This is how the notation is used. But, in this case, if you compare this and this - sorry about this - this is wrong - 9730 and 9498 and 9498 and 9730 so they're in both positions. In normal world, this should not be happening, you know? But we looked at it and then we figured out that both ASNs belong to the same company, right, they are two different parts of the same company in India. So that is fine. People do that, right. You have one business unit buying transit from another one and though business unit of the same parent company buying transit from the other guy. OK, we buy that argument.

But then this is another example where, you know, it doesn't make a lot of sense, because here is another example where we see 12069 and 23269 and here it is on the left-hand side and here it is on the origin AS side so who is buying and who is selling to whom? Right? You can't really say by looking at this. And our presumption is 7018 is probably selling to these guys. 12069 is not someone that 7018 and AT&T would peer with. So we find this relationship probably right and 12069 is probably selling transit to 23269, which means in the second case, it's very likely that 23269 is actually leaking routes. So they're working on and they are leaking routes on both sides or at least on one side that. This is our presumption.

To explain this further, what we did - how did we figure out the X relationship? Where two ASNs announce each other and how do we find that out? We did reiterative parsing of the routing table data from multiple sources. We ran data and as soon as we saw an anomaly we did further tests, cross-checked it with our own routes. We looked at peering routes at around 32 places with around 400 ASNs. We looked at what kind of routes they are sending. We've got sources in four locations around the world. US east coast, US west coast, London and Hong Kong, where you get full routes and we talked routes from people in Sweden, from Philip, and from LINX and analysed all of this and we are still doing that and this is how we came one the X relationship which actually doesn't make a lot of sense.

Now, then we said, "OK, let's try to figure out what kind of things people do," because the X relationship is very interesting because that is where the real problem is - you know, people can leak private ASes. People can, you know, mostly make mistakes, typographical errors and all that. But how do you fine those on a consistent basis? South so now we are working on using the X relationship, as we call it, which doesn't make any sense, as the biggest anomaly which we can further take and try and make sense out of it or tell people they're make ago mistake and people come back and say, "Don't teach us BGP. We are doing AS-path poisoning. None of your business." We expect everything.

Now this is an earlier example where two ASNs were part of the same company. Now, how do we find that out? Well, we use Whois. Whois is sometimes very useful, like in this case here. AS 10310 and AS 26085 both belong to Yahoo. We saw that these two had an X relationship. They're bot Yahoo. Put them in one bracket and they're done.

Then we looked at another two ASNs - 35324 and 35391 and they are not part of the same company and they're, you know, import and export relationship doesn't make any sense. They both claim they are selling transit to each other. So how useful is Whois there? It's difficult to figure that out, what is happening. I have some examples here that might be of interest to this audience.

I was intrigued by this particular one. I pulled this out today mainly because this might be of interest and maybe somebody can give me a clue as to what is Tiscali, predominantly a British provider doing between Asia Netcom and China Netcom. Tiscali doesn't make sense geographically or topographically unless Tiscali had a big network here prior to 1996. This doesn't make a lot of sense.

Now we can go further into this. We saw Asia Netcom, 10026, buying transit from MFN in the US, 6461 and we presumed - we don't know what the relationship is here because if you go and look fur, you that actually see that 3257 is also announcing routes on 10 o 026, but I don't have it here. 3356 is Level 3. So I can presume that between Tiscali and Level 3 they have some kind of peering and transit relationship, peering for Europe, transit for North America. And the same with Global Crossing. This kind of makes sense. This relationship too makes sense. And this one probably makes a lot of sense in saying that, OK, Tiscali might be connecting with China Netcom on a peering or transit basis somehow. But then this path here really doesn't make a lot of sense because, you know...or maybe it's true and it actually reflects the Internet. You know, who knows? Maybe somebody can explain that to me? Two providers in comparatively close geographical locations, with fibre overlapping each over are going through a British provider somewhere. That's something to look at. Again, this makes sense a lot more because Etisalat is here and Tiscali is here. This might be that 10026 is either leaking routes or I don't know. Our best guess here is that ANC is leaking routes. That's our best guess, leaking routes to other people and Tiscali is buying transit from these guys. But even that doesn't make a lot of accepts because Tiscali as a provider should not be sending opposite routes to its peer. Like, you know, it's slightly, yeah, I think... ANC is leaking routes. If somebody can explain that, that would be nice to see.

So you see a lot of this kind of problems and this one Avestan example I liked, especially in the context of this meeting and I brought it here but there are plenty of other examples where, you know, there is an ASN in LA which belongs to someone called Nuclear Fallout and they are announcing routes from Sprint and Global Crossing and MFN and I don't know what it is. I mean there is no data on this particular ASN. It just says Nuclear Fallout. It may be something. Who knows?

So the X relation count for this week, this is what I was doing this morning, it has been fairly consistent between 150 and 160 - 154 to 16 1 so we see a lot of these counts and we have not spent enough time analysing how many of these are actually typos. How many of these are AS-path poisoning techniques and how many of these are, you know, deliberate route leaks or some other thing. So we still don't know. That's hour plan - to look at it further and deeper and look at different views, regional views, global views, some regional views and so on and what we are thinking about doing is setting up an e-mail mechanism to report possible route leaks to ASNs. So if somebody wants to know if their ASN is one of those 154 listed AS numbers, maybe they can get an e-mail from us or subscribe to something like that.

The other solution we have from our friends in a big ISP was kindly set up a web front end where I can go and put my different AS, four different AS and see what kind of relationship you guise can come up with. And the other thing we probably would be interested in doing is looking at historical data. We've got archives going back to 1992 ourselves so one of these days I think we'll start running the test, reiterative test on archive data and see what kind of relationship we can actually build or produce out of this. So that was my presentation. Thanks. And if there are questions, comments, I'm very much happy to take it. We've got an e-mail address, BGP anomalies pch.net that you can use if you have specific questions or see something. Or you can also go to the APNIC website or our own website pretty much soon.

PHILIP SMITH: Are there any questions for Gaurab?

OK, while people are thinking about their questions, can I give some input?

GAURAB RAJ UPADHAYA: Yes, please.

PHILIP SMITH: In your Tiscali one, I suspect Tiscali are leaking everything they have towards Asia Netcom.

GAURAB RAJ UPADHAYA: But if they are doing that, Asia Netcom should not be sending Tiscali routes to MFN.

PHILIP SMITH: Can I pass to my co-chair? I won't put you on the spot. It's a strange one.

GAURAB RAJ UPADHAYA: I'm looking for input from providers in this region. We had a lot of direct feedback from providers in North America and Europe but it's been difficult to get incompetent put straight away from providers in this region and we want to get some more input on what we should be looking at or whether our approach is still valid. One of the things that came out last year was we did get on with the concept of regional tier-1s where France Telecom and Telia were the tier-1s in Europe and then there are the big 10 North American providers. So, that was our work last year and this was something else.

PHILIP SMITH: I can't believe there are no questions after all the questions we had this morning.

GAURAB RAJ UPADHAYA: BGP is not interesting.

PHILIP SMITH: Gaurab gave a similar presentation a few weeks ago at SANOG and got lots of questions. No? If not, thank you very much.

APPLAUSE

Last up, we have Merike who will talk about something completely different. IPv6, what works, what doesn't work.

MERIKE KAEO: Hello. So my name is Marike Kaeo and last year I had a really interesting project. I was working for Connexions by Boeing and I got hired to build a dual infrastructure for ground stations. I spent 11 months in the lab playing with 25 pieces of equipment, putting together a dual stack infrastructure. Unfortunately, they had a business problem. They decided to do away with it. But I have learned a lot and I want to help people, kind of educate them about what are the experiences I had trying to actually run an operation of a v6 network.

What I'm going do talk about is architecture considerations. "We have an address block, now what do you do with it?"; functional considerations, personal appointment experiences that I had; and then where do we go from here?

With architecture consideration, it was pretty interesting. We spent quite a long time figuring our addressing architecture. We got a /32, Boeing did, and we had these airplanes and they were thinking, "What if we have airplanes with a /48? What do you do with the ground stations?" One of the most interesting things was how do we address our point-to-point addresses.

I got a lot of views from meetings and asked people, "What do you do?" Asking 10 providers, I got three different answers. Generally, it ended up being in Europe a lot of people did say, "We use a /64." In the US everybody basically said to use /126. So it was pretty interesting.

There was one RFC that was written a while back, 3627 which offers guidelines of what to do and it says don't use /27 and it lists seven different things you can do point to point.

We ended up using 126s because we can have so many different addresses on an interface and you have multi-interface router. You really have to think about how do you actually name these things. What do you want to get access to? Do you want to do trace routes? Interfaces? DNS? You have to do some thinking in terms of how do you want to do your infrastructure. Do we do native routing verses tunnels?

I'm a security person, so I oppose doing tunnels. They always pose a security risk. Management - and I have a slide on that - management needs the most work. If you want to be native v6, because most of the management is done using v4 transport, security is the reason I did v6 years ago. Because I did not know what was meant by "security is built in". When I listened to people saying that, I thought, "Education needs to be done."

So infrastructure components, this is just a slide that basically shows what most networks have in some form or other. The middle piece, the customers have ISP and usually you have enough. These are the functional considerations in terms of, "What do we need to consider and look at?" And, "What are some of the issues, maybe, if you are going to run a dual stack for the network?" For that matter, also a native v6 network.

The router control plane - I was so surprised, all I did was configure it and it worked. Really it did. We were running IGP, and I configured the path filters. The provider that we actually used to natively peer with required MD-5. We did that fine with v4 and v6. So, with the routing stuff it was great. I know that in the last seven or eight years there was a number of ISPs that did the debugging for me. In terms of the routing things worked well. That was really nice.

Here is an example of the configuration with the addresses changed, and the lines in blue are the ones specific to the v6. You will note the last line is the password, for the MD5 with the occasion. It is the same configuration as for v4. Since I am an IPsec person, and v6 people should be thinking about that, I was saying, "People keep using v5 with authentication." It is a vendor issue. The vendor should make the configurations easier. What about having something as simple as the neighbour, IPsec pre-shared secret. That is all you have to do, instead of the configurations you need to do now. This is a continuation of the example we had. I had the defined prefix list and AS path filter so it says, "Permit everything in my AS", which is the one with the ^$ sign. We were only announcing our specific /48 to our PGP peer. The one thing that needs improvement in IPv6, and it is mostly is an issue if you were running OSPFv3, I couldn't necessarily do IPsec if I wanted to authenticate. That is my view of the problem.

One of the things also, I have a slide later on that, I will mention it now, is that in the standards body, when OSPFv3 was first defined, they said, "You need to use IPsec." Everybody that I know has implemented, if they use IPsec. Now there is a new draft that says, "How do I prospect OSPFv3?" And the standard mandates using it Null encryption. Different vendors may not be, probably.

Also, I ran into a problem where a couple of devices that supported OSPFv4 didn't support v3, so I had to play around with static. For the datapath itself, which is your transit path, basically what you do is you rate limit and you filter. Although it was configurable for v6, the logging in a lot of the products needed improvement. There were some products you could configure a filter, but you couldn't configure to log what was being done. I find that not very useful because I always want to log exceptions to a filter to see what might be going on.

Netflow is primarily used by most ISPs to track traffic flow. Mostly use v4 transport. URPF is usually available for IPv6. We have to consider, IPv6, it is not unreasonable that more encryption will be done. You don't want to punt and say, "We will get to that." Start thinking how it will affect your infrastructures, especially if you are looking at the data and doing analysis on it.

For a device in-band and out-of-band management, basically, most of the products were able to use v4 as a transport. All the majority of the other management protocols required that you use v4 as transport, which, in the grand scheme of things if you are running a dual stack is not a great issue, but if you are looking to native v6 it is going to be a problem.

There wasn't a product where it was officially supported, but it was in the code, but I would by far say in terms of management that is where the biggest work needs to be done to, is an operational IPv6 network. There is not enough traffic to really worry about it so much. But in the last six months, I have seen much more activity in the IPv6 operations and mailing lists and other forums.

This is going to be a concern. If you at all have any influence with any other vendors, help them make the appropriate decisions for their environment, for their part, list of what they are going to develop.

Software upgrade and integration, basically v4 transport is used.

So, general observations that I have is that IPv6 is being used and employed in many ISPs and in plumbing and routing, it works well. The majority of issues come from network services management and security. It does not mean avoid IPv6. More people should be testing it on equipment so they can look at what are the things that work, what are the things I need in my environment, and how can I influence the right people so I can get what I want.

The MS works pretty well, but you need better automation. I have talked to folks who use DNS and they are just starting to define their requirements for IPv6. Management works okay with v4 transport.

Security, it is basically built up, but you have to turn it on and have it supported in your products. It is not worse with IPv4. Monitoring tools need improvement.

Other observations. I started playing with the tunnel broker device. I heard so much how it was great. I seriously had this one-hour telephone conversation and one of the recommendations I had was, "Well, you really should allow me to use NTP." The answer was, "Why is it useful?" Okay, I found that rather problematic. Some of these vendors really do need to be educated, like the ones that say, "I understand IPv6 addressing and discovery, and how it works." That's great, but you need to understand what a product needs to do in an operational environment. This is where we need to educate each other. Basic filtering is configurable but you can't always log it.

There is no cohesive IPv6 support. One of the things I was very happy to see was that one vendor in particular, I used to be able to get IPv6 images, but it didn't necessarily have a crypto in it. I had no security. I couldn't configure IPv6. Most vendors have figured it is not such a great idea and I think in the future when you do have products and they have IPv6 support, it will, you know, you will be required to also have a crypto image. Should you use to use IPv6 you can.

There is a lack of debugging tools. When I was trying to figure out what was going on with v6, the debugging was limited. There were fewer devices. In terms of security principles, I do care about security. I don't know why, but I do. Don't do what we did before, which wasn't done deliberately, but it was how the networks evolved, but you have to think about it as an architect in the v6 network.

In terms of operator issues, I am used to certain commands with v4 and I thought I would use the same command with IPv6. It wasn't always the case. We have a lot of equipment and they say, "We have Cisco-like commands". It is not like Cisco - it is quite different. With IPv6, I found some of the things were quite different - enough that I have to, like, send a question mark to figure out what the command was, which, really, I don't like to do.

Here are some examples. If I have to access this in IPv4, in v6 it was "filter". People didn't talk to each other. I asked somebody at Cisco, I was there from '93 to 2000, I asked, "Doesn't the Parser police exist anymore?" It is just a matter of, "You change your script around. But, I don't know."

Standards fun, I mentioned the first one. I looked to see if your products, and you care about OSPFv3, if they do identification, if they are using null encryption. I dislike NAT profusely. It is a security issue. If people start doing NATPP you are going to have the same security issues that are caused by NAT. I try to avoid it. With IPsec, with those of you used to it, there was a v2 that became a standard in the last six months. It would be really nice if, on occasions, now you are required to use IP2, the issue is IP1 and IP2 are not compatible, not backwards compatible. It really has some optimisations.

IPv6 is not yet a standard. I see people talking about NTP and I realise is for UNIX.

NTP for v6 is not available and I guess the standard is not there yet. A couple of slides talking about IPv6 security. Really, people should divide security into any infrastructures they are starting to define.

Rule No.1 - don't break your working v4 infrastructure. No.2 - don't re-architect your current mess. But the security issues are really very similar with v4 to v6. But with v6 you have to have considerations, so you want to think about the considerations because you can have a bit of a more security infrastructure with v6.

Don't just map what you are doing with v4. It would not be optimal. My point of view, IPv6 security, all products need to support IPv6. People say, "I don't care about encryption". It does not mean confidentiality. In my view of the world, in the Internet of the future, all traffic should be authenticated and have security checks, everything. There is no reason not to do that. You always want to filter out the edges for sanity checks. They all need to support filtering and logging exceptions. You also have to have the auditing tools so you can see what is going on.

This is just a slide to show people, because even in the technical forums with really smart people, I see they don't really understand the differences with MD5 with IPsec. They are authenticating peers and, really, if you used IPsec you could get better and future solutions.

So with IPsec, what ends up happening is you used a shared secret or certificate to authenticate the peers to each other and they run an algorithm. This shared secret - it is the K there - it is a shared secret and then you use that to derive your authentication material.

The authentication material gets renewed again once a certain lifetime it is up. You are creating a new shared secret for running your authentication when you are doing say, BGP authentication. You share a secret. Until you touch the machines, it is not going to get renewed.

In some ways, using IPsec there is a way of creating new keys that you use to authenticate your writing updates. The issues with IPsec are that I have been looking at it since '97. There were issues in the beginning. The main issues today are that vendors use terminology where they use five different terms that mean the same thing. When you are configuring different devices, it appears complex, but you should only be configuring one or two lines and be done with it.

Here is my view of the world: some sample of future configurations. The reason I care so much, IP6, that is why security is built in. If nobody is using IPsec, you have limited security.

So where are we? I urge people to start looking at IPsec more. In terms of, if you want to do a realistic deployment now, you really and provide IPsec's capability. You have to look at the cost tradeoffs. A lot of the equipment, you can configure IPv6. You can also do incremental transitions. One of the things we ran into when we decided, "It works in the lab. How do we deploy it out to the operational ground stations?" There was a variety of ways. To start off, using a third-party tunnel program. There were a couple that were really well known and that way we could provide IPv6 connectivity. The second choice was to use our own tunnel broker and then run and operate it on our own and migrate over to native v6 as the operational cost became more viable and more customers could start using it.

So, this is just some of the things that need improvement that is critical for true large-scale operational IPv6 deployments. I have mentioned them already. The one interesting thing when I first did this talk, and I thought it was interesting, Geoff Huston mentioned it this morning about people pointing fingers at each other. I am sure you will hear more about it when people talk about multihoming, but I'm a neutral third party, I deal with ISPs, customers, and I go to the standards bodies. What I find is the vendors are waiting for their customers to say, the ISPs, "What is my priority list?", the ISPs are waiting for the users of products. Also, from a customer point, they are waiting for use of service from the ISP and they are waiting for the customers to give requirements and pay money.

The standards bodies are waiting for operational input to say, what do we do? Is what we are defining correct? The nice thing, about a year ago, there was a lot of finger pointing and everybody was in their own corner. But slowly, there is now work being done, more work being done, policies are changing, there is more work being done. I see a lot more work being done in all the communities, which is great. If we all, as a community, help the process, I think in whole, that the IPv6 deployments will happen with a lot less pain.

PHILIP SMITH: Any questions? A lot of information. The good people need time to think and they will discuss the questions with you in a coffee break, which we will now have. Thank you very much. It is now a coffee break, we are going to resume at 4 o'clock, so in 25 minutes.

I believe we still have one space for a lightening talk. If you are desperate to give a talk, please see the helpdesk staff now because we are going to close the doors on the call for the presentations.

SPEAKER FROM THE FLOOR: See Save who has the list there.

APNIC 22
APOPS - Session Three
Wednesday 6 September 2006
16:00 - 17:30


PHILIP SMITH: OK. Let's make a far... let's make a START to the final session.

(Laughter)

I was going to say, "Let's make a final start." Before we begin the final session, I just want to tell you - a quick reminder about the APNIC social event. The buses will leave here starting at 6:50 and then every 10 minutes until 7:20. One or two of you asked me, "If I stay for the lightning talks, will I miss the social event?" No, you will not, because I am staying here to chair the lightning talks and do not want to miss the social event. The buses start leaving at 6:50 at the side entrance of the hotel lobby. You need your ticket. If you don't have your ticket, you won't get on the bus. The last bus leaves here at 7:20. So there's plenty of time for you to go back and change into party gear or whatever before you catch the bus.

Now, on to the final session, quite a busy one. More interesting topics, I hope. First one up is Geoff Huston, who will be giving an update on the routing certification project. Geoff.

GEOFF HUSTON: Thanks, Phil.

This is a quick update report following on from a number of previous presentations about the trial that's going on in APNIC on certification of IP addresses and AS numbers. Very quickly to recap on the motivation, obviously there are many, many, many ways to be bad on the Internet. One of the most insidiously cute ways to be bad is to corrupt the routing system. If you want to be bad and don't want to be caught, changing the way packets fly on the Internet and intercepting them in places they weren't meant to go is a remarkably insidious way to attack and quite frankly our defences are not working - bogon filters, route policy databases and so on are not an entirely robust form of defence. So we need to do better. And if you go back to the basic - what are you trying to answer - some basic questions. Is that address prefix a real one?
Did someone invent it? Who is injecting it? Do they have the necessary credentials? Did they really get that address? Are they authorised to advertise it? There are many ways to answer that. A lot of the ways are expensive. Expensive is generally a barrier to deployment. It needs to be quick and cheap. Is there a way to answer these questions in a fashion that is reliable, quick and cheap?

Wouldn't it be good if we could reuse the existing technology with public key infrastructure to answer these basic questions? Can I as an address-holder authenticate someone to advertise it for me? Could a recipient of the address understand that it is really me, it really is my address and I permitted it to be advertised in this way? Did someone tamper with that information?
You'd like to know if they did. You're talking about trying to use PKIs to convey authenticity in the routing system. That would be really cool if we could do it.

In the trial, we've been experimenting not with the routing protocols, not with trying to develop a secure version of BGP. Before you get there, you need to understand how to do the basic credentials of trust associated with IP address resources and AS numbers. In this trial, we're actually looking at certification. And in so doing, what we're trying to do is to use some relatively simple approaches that do the job. Firstly, we're trying not to invent an entirely new raft of technology. Use existing PKI technologies as much as we possibly can and one step further - actually try and leverage existing open source software tools and deployed systems, trying not to invent too much of the wheel here in doing this.



I believe that this is a lot of work for everyone and keeping the results secret and propriety and does not help. Open source solutions appear to be the appropriate way in this community to contribute into the solution. So we're certainly looking at augmenting the open source solution set with our contributions. And, yes, trying to do this is also part of trying to create industry standards that support authentication of IP resource information. So we're certainly trying also to contribute to open standards wherever possible. So the overall approach is to use existing X.509 Public Key Certificate technology with the extensions defined to support IP address resources and the open source tool kit we'll be using as the foundation is one called OpenSSL and we're leveraging on that to produce a suite of tools to allow you to talk about is this address real?

Did the address-holder authorise this address to be routed? Has the information been tampered with? Is this what the address-holder actually said? Resource certificates are actually pretty simple things. If you look at a certificate, what it really says is the issuer said that the subject of the certificate whose public key is in the certificate is the current unique controller of a collection of IP address and AS number resources and they're listed in the certificate. This is a simple right of use that can be used to authenticate that person's subsequent addresses.

I don't actually say, and the certificate doesn't say, whether the recipient is good, bad or evil. If I issue a certificate to Philip Smith, it doesn't really mean that Philip Smith is going to be a good guy thereafter, but what it does say is that Philip Smith has that unique right of use of those resources, and that anything Philip Smith does he can sign with his key, and that you can look at anything Philip does and validate the key signatures and validate it really was Philip who said it and it really is those addresses that he's talking about.

What can you do with them? The standard things you do with public key infrastructure. Philip now, as the holder of this key and certificate, can now sign things, he can sign routing authorities. "Please, Geoff, route this address." I go, "Well, who are you?" "Well, I'm Philip and I own that address. I'm the current holder. I have signed the request. If you validate that, you can understand it was really me that issued that request." So routing authorities, routing requests, even Whois objects or Internet Route Registry objects, can be signed with your private key and anyone can then look at this and validate that that is true and authentic information reflect ago valid address.

You could also, if you were a Local Internet Registry or a National Internet Registry, as a holder of one of these certificates, issuer of these certificates, when you issue addresses further downstream, you can accompany that with a certificate that attests to that action of validation or assignment. And anyone - you, me or anyone else - can validate what's going on by looking at the assigned objects and validating the certificate. Did the resource holder really do this? Is the thing that they did precisely what I'm seeing? Has anyone tried to tamper with it? And is that still valid today?

So in looking at this, there's a lot of complexity inside certificate infrastructure and there's an awful lot of technology. It's certainly not, I suppose, our intent to try and make everyone and every single member here and all the other scenarios with address resources having to have them do a complete certificate infrastructure of their own. If you want to, sure. But it's not necessary. Certainly as part of this trial, what we're trying to do is to create a service interface via an APNIC web portal that does most of the back-end work around certificate management on your behalf so being able to generate and sign objects is part of our objective here. To be able to validate objects against this PKI and even to manage subordinate certificate issuance. Our suspicion is that a lot of people would use it in that way in the way they use MyAPNIC to manage resources.

For various reasons, there are folk who want to manage their own keys and their own key-based activities and that's fine. We'd like to assist in generating some tools you can use to allow you to manage certificate repositories, to manage your own signing of resource objects and to generate and lodge certificate objects. So that's the general thrust of what the outcomes of this activity are intended to be. So what does it look like? What does it mean?

I want to sign an Internet router registry object. If that's what you want to do, that may be the kind of thing that you would generate. What you see there is an example of a signed object. The first part should be relatively familiar. They're a standard rout registry object. There are two parts there that are highlighted. The first is the reference, an rsync URL that actually points to where the certificate is where the public key can be found. So sigcert is a URL that says that certificate, that public key in there, will match the signature block of this signed object. And the note below that is the signature, the sigblk, which is the encoding of the encrypted message of the rest of the object.

Here's the certificate that matches that. So this is the certificate that says this is a version, this is all these wonderful fields. The key thing is there that the subject key identifier is identified in here and should be somewhere a subject key. I think I left it off because it was too big but that's a certificate that would have been used to sign the object.

I'm not sure that that technology is terribly useful per se and what we're trying to do is to actually abstract that and give a simpler interface via a portal that looks at this in a different way. What are you really trying to do in using certificates? And the answer is you're trying to sign things. You're trying to sign resources. So with a portal that looks at this from a collection of resource signing activities so that you might have a collection of resources and in very small print there you might have an object called 'all', all the IP resources you hold, all of the AS number resources and so on.

What you might then do is use those to sign documents, to sign objects. Whois objects, route registry objects and similar, so that there would be a section there that manages your signed objects. So the documents are signed with resource collections and all of the underlying mechanisms and machinery to manage the certificate are effectively happening as a background process. What you see are resources and signings.

How far have we got? A fair way. We spent a fair amount of time making sure our specification for resource certificates is clearly understood by all of us as well as everyone else. So that specification of X.509 resource certificates is currently inside the IETF on the standardisation track. We've actually been generating various resource certificate repositories on a trial basis, looking at if we had to actually generate certificates to cover all of APNIC's allocations would what it look like? What would the repository structure look like. We've spent some time generating the certificate repositories and manipulating them. More recently, we've been developing tools to be a authority, being done by Robert Kisteleki from the RIPE NCC looking at how you interact between the certificate authority and the resource authority.

The work is still going on at a relatively frenetic pace. We trying to make sure that OpenSSL, that open software suite, is extended to understand resource certificates.

And this activity being done by Rob Austein is being supported by ARIN. In APNIC, we've been working on tools for resource management, object signing and signed object validation. We're looking at LIR and ISP tools for certificate management and spending a bit of time looking at what does it mean for the operation of APNIC, what does it mean for you as users of such a system.

This is a trial rather than production service. There are many things, I think, that we need to consider at the end of the trial before we move to production. So the next steps are to complete the current trial activities across the last few months of this year and then undertake an evaluation and review of what we've done. We'd actually like to understood that were our initial assumptions that PKIs really are the right way of doing this a good approach? And then have a look at the implications of this form of certification of resources - what are the implications in terms of service infrastructure, operational procedures? How useful is it? Is this really a reliable, cheap and useful methodology for you to use in order to make the routing system more secure? in order to detect lies and bad things in routing?

And resource certification also has some issues in the policy agenda. What is a certificate exploration? What does revocation mean? What is the service model? There are some implications from a policy standpoint that we, as a community, will need to look at. This review will try to highlight some of those issues and assist you in understanding what kind of policy changes might be appropriate in a certified world. And also of course having a look at what would it mean for APNIC to run this as a full blown production service, as a service to you as the membership and beyond that into the wider community. What sort of things need to be done for production deployment. That's where we are right now in a relatively quick snapshot. There is still work to go and we certainly would expect - in fact I do expect to do a relatively comprehensive report to you probably in written form by the March meeting next year as to what we found out in this trial and what the next steps might be.

So I think I'm on time now. After any questions I'll be happy to take them. Thank you.

PHILIP SMITH: Any questions?

(Pause)

The lunch has been far too good. It's changed all the hyperactivity from the morning into stunned silence.

OK, well, thank you very much Geoff.

GEOFF HUSTON: Thank you.

PHILIP SMITH: So the next speaker is Taiji Kimura from JPNIC. He's talking about route origination authorisation with the IRR.

TAIJI KIMURA: I'm Taiji from JPNIC. I will give a short presentation about ROA with IRR. OK. I will give a presentation. And as shown by Geoff, the project is being coordinated by APNIC and RIPE NCC. After the last IETF, I felt we are facing the next stage that we can discuss on personal issues about that.

What I want to share today are two of my ideas for management logics of resource certificates. One is use of IRR for handy and legitimate information for resource certificates and another is a new model, it's not quite new in the PK I meaning - external RA model for simple deployment.

This is a very short presentation, so I have only one agenda - and this has one point of view.

After the deployment of resource certificates, router operators will have additional works, at least these two. One is managing certificates issued by registries and the management of certificates according to ROAs.

ROA - route origination authorisation - means authorisations from LIR to routers to AS, authorisation given by LIR address space-holders to AS to originate routes in the Internet.

For managing resource certificates from registries, it's better to be issued by a CA who has not only legitimate means, legitimate allocation and assignment, but also has handy information.

Legitimate allocation and assignment are from RIR and NIRs, but handy information - where is handy information? It's registered in IRR. Because IRR has more informational allocated but these informations are managed by ISPs by themselves. So this is a very important factor for real deployment.

Focusing on IP registries structure, we can get the legitimate bindings about LIR's information, about the LIR's contract and what information the LIRs have and address prefixes, which address is allocated to which LIRs. And, of course, we can get AS number information.

And focusing on routing registry, we can get handy bindings between AS object and route object. This means that - which AS may announce which ranges, registered as route objects and from the LIR only the legitimate bindings, we can find if they are useful or not.

But if the resource certification looks to the routing registry, we can get useful and legitimate information from both systems. This is one of the ideas in my presentation.

This is for managing ROAs. We can say one thing - a whole range of address that one AS can advertise is for larger, for bigger, than address ranges that one certificate has. If all certificates have all allocated ranges then when new allocation or new assignment effects all certificates, they are managed by one AS. So router operators will need to manage prefixes in certificates according to ROAs. So this requires further certificate management system for personal use.

And we can see the complexity near the end of tree - tree means that PKI tree structure for resource certificate. We have the tree structure and so we should have the implementations for each entities, like LIR CA and ISP CA and who issues the resource certificate for subordinate CA or for use of routers.

But they have different functions according to their roles. This picture shows two things. For LIR CA, they arrange certificates according to ROAs and allocation from NIR are RIR. And ISP CA will arrange their certificate with routers configurations and ROAs.

Then external ROA will simplify this. External RA is external registration authority and this can make it easier for ISPs to manage their certificates. So ISPs can manage their certificate by themselves. But these operations are restricted by LIRs according to the allocations.

Then, I have shown two ideas - one is use of IRR for resource certificates for handy certificates. And external RA for ISPs - this is for simplified deployment.

This is - I want to show that, well, that APNIC can handle or RIPE NCC can handle - then I thought that both of them can handle this model on implementations because they already have both systems - IP registry system and IRR - and of course we have members' certificates for authentication to access to MyAPNIC, then external RA model can use this certificate.

That's all for my presentation.

PHILIP SMITH: Any questions at all?

GEORGE MICHAELSON: George Michaelson from APNIC. I'd just like to say thank you for that presentation. I think many ideas you've presented here reflect thinking that has been going on inside the design group in APNIC and with the other RIRs and that it's really, really gratifying that we have convergence on a common understanding of behaviour in outcomes. This is very good. Thank you.

TAIJI KIMURA: I am very happy to hear that.

TERRY MANDERSON: Terry Manderson from APNIC. I have one question. In the concept of the IRR, how do you expect to maintain currency of the records in the IRR? One thing that I notice as a long-time user and operator in the Internet routing registry is that data tends to become stale because the data is managed by the ISPs and by the LIRs. So what motivations do you have and what motivations can you provide to those people to maintain the current data? Whilst in that same vein, I also notice that many IRRs have similar data as well as conflicting data - which IRR do you take as the authoritative point for the data? A couple of questions you might like to consider.

TAIJI KIMURA: OK. Yeah, it is very difficult work. It brings difficult works. But I think two ways to clean and make route operators to have motivations to clean database is - one is cleaning just the data, registered data, for comparing with route objects between the route object and we can work by using the rest, just for cleaning that data.

And the second thing is from the beginning, registration - from the registration, we can check the route objects with allocation database if we have both systems and of course we can authenticate the identity of the user who wants to register some objects. If we take two ways to clean and to have motivations to - router operators to clean the databases, is that the answer for that?

TERRY MANDERSON: That's fine.

PHILIP SMITH: OK. Do we have any other questions? No questions? OK, thank you.

OK, next up we have Merike again who will be talking about ISP security practices this time. This is called getting value for money or something.

MERIKE KAEO: Hello again. So this time I'm going to be talking about operational security, current practices. About two years ago, in the IETF they started an operational security working group which was basically to look at what capability and functionality was missing in current operational networks, either from devices, also practices. And one of the things that I did was talked to most of the ISPs about some of the ones that are in this talk. I personally went and talked to their security people. They gave me a lot of information, most of which I can't really talk about and then I had to kind of generalise it into a document so that it would help educate people in terms of what is it that ISPs are currently doing for security and where are the issues. And, you know, I think all of us who work with or have ever configured any of the security functionalities know that, you know, it's far from robust, right? We're just trying to do the best we can.

And I failed to mention it in my previous talks so I'll mention it now. I recently finished a paper for the IPv6 forum which is an IPv6 security technology paper and if you go to www.ipv6forum.com there is some information on that which might be useful.

But anyway, this again is the same picture I showed in my other talk which just shows you what infrastructures basically look like.

And so I'm going to concentrate on, you know, what are the findings from my survey as to how do large ISPs project their infrastructures. You know, the first thing is that everybody has to understand the problem. There is a lot of people that I talk to, especially from smaller ISPs, where they end up saying, "Oh, yeah, we filter." And I'm like, "How do you know what to filter?" and they're like, "Well, um, my buddy told me what to put in." And you have to understand your networks and what's important and, you know, it's really important to understand what the security issues actually are and the risk trade-offs.

I gave a talk or a workshop earlier this week and I always ask how many of you actually have a security policy. How many have read it? And you know, security policy - people hate policies, right? I mean, if you're technical, it's sort of like, that's what somebody else does. But really you need to have some kind of a policy to then decide how to effectively, you know, implement technologies. And you always wan to have procedures in place for incident response. There's no such thing as absolute security, right? Most people don't talk about it. Usually everybody has had some kind of a security incident and usually the first time you have one everybody runs around going, "What do I do? It's your problem. I don't know. Who do we call?" at best, even if you haven't had a real serious problem yet, at least to understand what people are responsible for and all of the large ISPs obviously do have this in place.

In terms of understanding the problem, one of the issues with security is that people use so many different terms to mean the same things that it always seems like it's such a huge and difficult problem whereas to me it's really simple. It's all a matter of access control. Knowing who has access to what and then what are they doing. You know, that's all it comes down to. And one of the things we have to look at - what are the possible attack sources? Right? Somebody who is trying to harm your network or divert your traffic. So I categorise them in four different ways. One is you can have passive versus active attacks. So you know active attack meaning that you're actually putting packets on to the wire. Passive is that you're looking at what's on the wire or in the ether.

On-path versus off-path. Meaning are you directly in the data path or are you somewhere else and injecting false information, i.e. like false routing information to divert the traffic. Are awe trusted or untrusted entity and, as networks change in terms of perimeter changes, who are trust and who you don't trust is a grey area these days. The last point is one that really needs to be understood. A lot of - especially in the routing world, a lot of issues really are not always due to deliberate attacks, they're doing to configuration errors, but what, you know, it's destructive to the network regardless of whether or not it's intentional or unintentional. So, you know, typically you do have to look at all of these forms of attack sources and figure out whether or not that's an issue in your environment.



And really we don't need to care about the attack sources so much. What we care about is the impact that would do to our network, right? If there is zero impact, what do we care? And these are the four general categories and I took these from a standard which is RFC 2828 and, you know the terms basically are - you know, these are not commonly used terms but unauthorised disclosure just means that information is getting out that, you know, unauthorised people are capable of seeing. Deception is basically impersonation. Disruption you this of as denial of service and usurpation is getting access to devices you should not have access to.

So the security services that everybody needs to think about are as follows: There is user authentication and authorisation, data origin authentication - and I always make an effort to differentiate between user authentication and data origin authentication. People very commonly say, "Oh, yeah, that device where I'm authenticating to so-and-so." What does that mean? Are you validating the IP address is what you think it is? Or are you actually validating that I, who is using this machine with this IP address, am who I say I am? And typically you really should have a combination of both user and data origin authentication but, you know, it's pretty important to make the distinction between the two of them. Access control is basically filtering. Data integrity means that you're pretty certain that nobody has changed or modified the data as it's gone from sender to recipient.

Data confidentiality is encryption and auditing/logging and DoS migration.

So awful these security services I talked about with these ISPs and looking at all the different functional considerations. So, you know, as I went and talked to the first couple of guys, we're like, "What we do in security is huge so how do we actually categorise this?" and they said, "Well, the easiest thing for us to do is what do we do in an operational environment?" so the functions we have is device physical actes, management, in band or out of band. Data path which is transit, what do you do with the customer traffic that comes in and out of your networks? The router control plane and software upgrade and configuration integrity. So with all of these functional considerations, we talked about the security services that they use or don't use and why.



I also separately in this survey document talk about logging, filtering and denial of service for tracking and tracing. Logging and filtering is covered in all the functional considerations on the left but because they're pretty important categories in their own respect, it was, you know, I felt it was important enough to just reiterate some of them and some of the practices that are used.

So if we look at the device physical access, I'm looking - it's a surprise that equipment is kept in highly restrictive environment. Now, you know, I had a conversation with somebody yesterday and they mentioned that it was interesting where they have seen one environment where they had basically a Microsoft, a domain controller in a very highly restrictive area but the secondary controller, which had the exact same replication of information was actually sitting on somebody's desk. Right? That's not a good idea, right? You want to think about what, you know, if somebody has physical access to any machine that is part of your infrastructure, what's the damage that they could do? And it's probably more than you would think. And what was is it - I guess in Australia, right, about a year and a half ago where people impersonated some maintenance people and carried off a mainframe.

This is happening today. It's not like this happened 10 years ago. People always think that these security issues, you know, we've got to have the best encryption possible and really, I mean, if I want to go and hack into somebody or cause some damage, I want to make it easy for me. Right? And so you have to think about these things that you think are simple but it's amazing how many people just leave things - you know, don't do the simple things.

Console access is always password protected and access is more often than not using out-of-band management. Now remember these are big ISPs. You know, there was a couple of people that commented on this document saying, "Do people really do this?" Remember these are large ISPs. If you're in an environment where you're smaller and have more cost considerations, you know, you may not be able to afford the extra infrastructure to do an out-of-band management system but again that's up to you in trade-off in terms of how much risk do you want to leave yourself open to.

Individual users authenticated and social engineering, you know, this is a real important thing. I can't even begin to tell you how many times I'm standing in elevators and there's, you know, people talking to each other and saying, "Hey, I forgot my password, can I log in?" and I look around and look at their badge and how quickly can I figure out what their IP address is might be and hack into their system. People have to be taught to talk -- not to talk about things that are confidential, especially passwords into devices.

In terms of management, logical management of these devices, and logical management means, you know, your SNMP access, your web-based access, your Telnet SSH access. In-band and out-of-band management, there's basically similar security services. It's just whether or not you are using the network, whether or not you're use ago separate network to manage these devices or if you're using the same network as the data, the transit data is going through. SSH is primarily used. Telnet is usually only used from jumphosts. There are situations where people have to use Telnet because they have legacy devices. But you would never from this conference Telnet to an infrastructure device. You would SSH to some device that was in your trusted domain and then Telnet from there and that is always done.

So that at least you keep, you know, the passwords encrypted. All access is authenticated. The password mechanisms vary. You know, people use token-based passwords using a gold card or S-key or whatever.

AAA, the authentication authorisation and accounting protocol, is always used because it's just there are too many users and you always have a single local database entry for backup so typically within any device you can say user name, staff, password, something and if the policy for these ISPs is to use AAA and have your authentication offline somewhere, you don't want to leave yourself open for never having access to a device if that server is unavailable for whatever reason. Right? You always want to think of a backup. Anything that's on a network can potentially not be available so what do you do in those cases? One thing that I - I mean this is not something that these ISPs do but one thing that I do see and I see this in vendor documentation is if you can have a backup mechanism, some things they say, OK, log hip, authentication, radius and then use backup is none. I think that's the most ridiculous thing I've ever seen. You can always use backup mechanisms of using the internal database if you ever say none, then you have zero authentication whatsoever and I've seen that in a lot of vendor documentation. So, you know, be careful of just using things like that.

For management, people get lazy about this but you should always have filters in lace for your SNMP, your SSH host that you allow to get in because it just helps alleviate some of the risk. Obviously IP addresses can always be spoofed but somebody has to try harder. So if you are in an environment where you know what machines you communicate with for SNMP or NTP or whatever, or you know what machines they'll SSH in from, put in the filters. All devices do this. It's important in their environment and it's good practice.

Out-of-band management is basically the same security services as for in-band. The only different is that you have a separate network devoted to getting access to your devices. As backup most people use dial-back modems and usually dial-back encrypted modems.

The data path refers to customer traffic and really ISPs don't want to be the Internet police. They have to in some way, shape or form because they need to make sure that their own infrastructures are up and available so filtering on the primary mitigation techniques for security issues. BCP-38 guidelines for ingress filtering are primarily used. BCP-84 is the one that was an update to that to deal with multihoming issues. Some of these larger providers also do null-routing and black-hole routing for any malicious traffic. And there's also denial of service mitigation. Netflow is the primary method used for traffic data traffic flows. URPF is not consistently implemented and that's been kind of interesting because there are some varied schools of thought about, you know, how useful it is and whether or not, you know, it's worth while configuring these things.

Everybody logs exceptions.

The routing control plane was I think the most interesting one, where MD-5 authentication is sort of used in some places. Some providers require that their eBGP appears, use it. Quite a few just wait for the customers to ask for it. And so it really all depends on if you're a believer that this will help or not in your situation. Route filters are pretty much used and packet filters. Which filters you use and how you use them varies quite a bit. Prefix filters were used quite a bit. There was a survey that was done a while back and I note it just on what people do for filtering and it mimicked basically what I found as well is that more people are starting to use AS path filters and that's because they're easier to set up with prefix filters, you know, the list can be really large and it can be too taxing on your devices.

Route dampening, you know, was thought to be a good idea a few years back but as of I think two years ago research was done and it was decide that really it probably causes more harm than good.

In terms of software upgrade and integrity, you know, files are stored on specific systems with limited access to all of these hosts. S CP is used where possible and FTP is never used. What's interesting to me is that FTP is always used for downloading configuration files and the one thing I fine interesting is that I am a believer of IPSec and why people are not configuring IPSec to authenticate or encrypt this kind of software traffic I don't know because everything is in the clear using FTP so a lot of providers said they realise this is an issue. They basically log and monitor much more, you know, these out-of-band management systems. So if there was something simple, right, I mean, yes, IPSec is hard to configure but they are looking for some solution. If a simpler solution existed, they would want to use it.

Filtering considerations - there was a separate section that I used for that and basically people look at either the routing control plane filters, the data forwarding plane which is, you know, traffic going from one customer to the other or your management plane which is, you know, filtering your AAA Syslog as an SNMP server, mail server, DNS server, things like that' guess the only thing I want to say is you have to look at all three of them and there's more detail in the document itself.

In terms of denial of service, denial of service attacks are just - they're hard. And these large ISPs. It was really funny when the first guy talking said, "Yeah, you know, we have these attacks all the time and if it's less than one or two gig, we don't even look at it." I was like, "You're kidding." It was like, "We don't have the time for it." And it I was like, "OK." It taught me that, wow, you know, there is so much out there. I have a home router and little subnet and my Home Office and all that and I just kind of started looking at what people are trying to do and it's absolutely amazing. I'm just at home and I'm logging exceptions and there's scans happening at an unbelievable rate. Of course me too. I'm looking at these log files going, "I'm not going to try and figure out what's going on.

I've got other things to do. I've got to go out at night." It's amazing. Denial of service attacks - Botnets with millions of posts on them coming from all directions. So a couple of years ago, a bunch of these security guys at larger ISPs came up with interesting techniques to automate trying to basically filter traffic that's part of these large denial of service attacks. So one thing is to create sink holes where you take this data that you have identify as being part of a denial of service attack and then, you know, possibly divert that to some kind of analysis network.

Another thing - and this is a pretty widely used technique but it is fairly complex so it might take a little while to really understand what's going on - is it's call black-hole triggered routing and, in effect, what you do with this is you use a combination of Unicast Reverse Path Forwarding and routes in BGP to be able to apply filters to your filters as quickly as BGP can provide updates and you end up using Unicast Reverse Path Forwarding which, for those of you that may not be familiar with it - what it is when a packet comes in, it looks to see whether or not a source prefix is in your routing table or forwarding information base and if it is, it decides it's got to be a valid source and forwards it. If it's not, it decides it might be a spoof packet and drops it.

Basically that's the slide.

Let me go back here. I don't have any slides that specifically show black-hole triggered routing but my slides from the tutorial that I gave on Monday are actually online and I have four slides that show in a little bit more detail how that's used and if you have - if you are, you know, running an ISP and you have a lot of, you know, you're fairly large and by large I'm talking like 100 to maybe 200 routers, and you have issues with denial of service attacks, you know, come and find me and ask me a little bit more about that because it's a technique that may be useful in your environments.

I also asked people about IPv4 and IPv6. You know, you have the exact same considerations for IPv6 networks although some of the tools are not yet there for IPv6 transports. The biggest issue with IPv6 is that there aren't a lot of people that are tunnelling. Any time that you're tunnelling, you leave yourself open to security vulnerabilities because you do not always know who the end points are. Flow connection tools are not yet capable of detecting much malicious traffic.

That is going to change.

So as a summary from what I found in doing the survey is that the risk mitigation techniques are similar, yet different. There are similar conceptual safeguards. The differences are based on performance issues and operational complexity. One interesting point was why somebody wasn't using Unicast Reverse Path Forwarding - or rather they were using it on all devices where the interfaces were DS-3 or below and they did that because all of the devices that they had, uRPF supported that, but they had a number of different devices or different types of devices from the same vendor and if they had higher speeds than DS 3 then uRPF wasn't necessarily consistently available on these devices. So rather from an operational perspective, you know, telling these people that on these devices and those interfaces you configure it and on these devices you don't, it was much easier from an operational perspective to say on DS-3 and below we do uRPF and on others we don't.

That's something for vendors to know, that if it's a useful feature, it should be used.

Another one of these things was the TTL hat which, you know, people were very excited about three years ago and now recently it turns out that really many people aren't using it and the reason why they're not using it is because it's not consistently implemented or the interoperability doesn't work that well between different vendors. So although it could be a useful security feature, the capabilities in devices isn't - the implementation isn't good enough to really be useful.

So that's why this particular working group, there's other documents being created to specifically discuss the capabilities that are required so that the products can be used more effectively for secure deployments and if any of you are interested in that work, you know, please look at the documents that are existing currently in the IETF and the operational security working group because contributions from the operator community is, you know, definitely wanted.

So thank you and if anybody is interested in the text, it's actually number 7 and it recently passed the working group last call so it's more or less final. That's it.

PHILIP SMITH: Any questions for Merike?

Where is the roaming mic?

GEOFF HUSTON: Geoff Huston, APNIC. I often wonder a lot about the difference between security pantomime and true security.

And in thinking about the kind of mechanisms that happen in terms of real attack, and the responses in terms of infrastructure responses inside a network, it always strikes me that there's a whole bunch of being seen to be doing something because you think you should be doing something as distinct from really trying to attack the problem itself.

For example, black-holing packets with invalid source addresses because the source address was invalid, the attacker never wanted a response anyway, so the obvious thing is to go and steal other people's source addresses and launch the attack anyway. So isn't black-holing those kinds of packets with invalid source addresses an instance of pantomime rather than attacking the real problem? The real problem is a little bit more insidious than that.

I wonder how much of this scales. I wonder how much of this only works at low speed with lots of silicon and as we build bigger and bigger networks, if we stress the silicon, we don't have answers any more. Is the problem truly one about end systems and the long-term observation that infrastructure can't tell the difference between good packets and bad packets. They're just packets.

MERIKE KAEO: Yeah. I think that's a very astute observation, right? I mean the problem that the ISPs have had forever really, is that they act as the Internet police whereas really the end systems should be held liable for bad stuff that they're sending, regardless of who it is. And in terms of the black-hole routing, it started off where it was destination-based black-hole routing where you're dumping all the traffic from a specific source address or a number of different source addresses. Then they ended up doing more clever things in terms of looking at source and destination addresses and so there's always these tweaks that ISPs are doing trying to discern what is legitimate traffic versus what is really in effect bad traffic because, as Geoff alluded, you know, a lot of attackers, right, maybe their intent is just not to lot certain source traffic through, right?

So security from an ISP's perspective, really, ISPs should just pass data, right? And, you know, this is where filtering is really the most critical thing that people can do and if, you know, I deal with both enterprise people and service providers and enterprise people really do need to be much more responsible for the traffic that they're sending, you know, I don't know what the solution to that is and as ISPs you have to make sure your networks are available and so you do the best you can.

PHILIP SMITH: Any other questions?

Nothing on Jabber?

OK. Thanks very much, Merike.

APPLAUSE

OK, so next up we have Yoshinobu Matsuzaki from IIJ.

YOSHINOBU MATSUZAKI: This is Yoshinobu Matsuzaki. We have many considerations in security area but today I direct my focus on IP source spoofing. What is IP spoofing? Creation of IP packets with source addresses other than those assigned to that host. Many networks use IP spoofing at this moment. It is open to attacks. These are some concepts of malicious use with IP spoofing. Impersonation, hiding.

First is impersonation. The sender sends packet to victim. The source address of the packet is the address. The packet reaches the victim and the victim believes this packet came from purse partner and then the victim processes the packet. This is useful for data injection. This impersonation is used for spoofing as well.

Second is hiding. The sender is just hiding himself from investigation. The victim cannot know the real origin of the packet.

The third is reflection. The sender sends IP packet, IP spoof packet to the victim. Then the reflector sends the reply to the victim. So the victim sees a lot of replies from reflectors without any requests.

So attacker uses reflection for attacks. There are two famous reflected attacks. One is smurf attack. This is spoofing and the ability to broadcast as an amplifier. Another one is DNS amplification attacks. This uses IP spoof reflection.

What is amplification? One packet, multiple replies. If sender sends one packet and gets multiple replies, this is the number of packets amplified. The smurf attacks use this type of amplification.

Type 2 - bigger reply. If sender sends a packet and gets a bigger-size reply, this is also amplification. This is an application of this.

IP reflected attacks use IP spoofing and open amplifiers. The attacker sends a lot of packets to the open amplifiers. The open amplifier sends amplified replies to the victim. It saturates the victim. The attacker uses amplification to increase the impact of the attack.

This is smurf attack. The attacker sends IP spoofing to a connected broadcast. These hosts send an echo reply to the victim because of the ping. The ping is the victim's. The victim receives tonnes of echo replies.

This is DNS amplification attack. The attacker sends IP spoofed DNS queries to the server. The source address is spoofed with victim's. Victim receives tonnes of DNS replies.

This is more realistic attacks. The attacker uses Botnet to send a lot of inquiries at once. He also sends huge DNS queries to hijack the server so he can get good amplification ratio during the attack. In this case, the example servers, who are compromised in some way.

Then we need solutions. One is disable the open amplifiers. If the amplifiers do not answer for the created from external, the attack packet can be dropped here.

Another is prevention of IP spoofing from every network. If attacker cannot use IP spoofing, IP reflected attacks does not work any more.

What is disabled open amplifiers? This depends on attacks. To reduce smurf attack, we have to disable directed broadcast. To disable recursive attacks we have to disconnect DNS server. It should receive from its customer only. Has to stop unnecessary queries as well. Prevention of IP spoofing - so this is source address validation. There are already good reasons. Source address validation is checking the source IP address of IP packets.

There are three points. First, filter invalid source address, of course. Second, filter close to the packets origin as possible. Third, filter precisely as possible.

If no networks allow IP spoofing, we can eliminate these kinds of attacks.

Why close to origin? We can check and drop the packets which have unused address everywhere. For example, if the packet has a source IP address 0.0.0.0, we can check you are spoofing. But the space can be checked before application. In this case, RT.a cannot determine if this is spoofed or not. So we have to prevent the source address validation close to the packet of origin as possible.

How to configure? We can use ACL and uRPF check. ACL is a packet filter. uRPF checks incoming packets using a routing table. It looks up the return path for the source address.

There are three more. First, is loose mode. This mode checks only if the source address is routable or not. We cannot stop the IP reflected attack with this mode. So, we have to use different modes to stop the reflected attack. It checks if the incoming interface is matched with the source address for the return address.

This is a sample configuration, ACL for Cisco. We have to configure ACL's first and then the interface. This is for Juniper's. We configure the firewall first and then to the interface. This is uRPF mode connect router configuration. This is uRPF configuration for network.

This is IIJ's policy. We use stricter mode for our single-homed customer. We using precise mode for customer mode.

Conclusion. This year is deterministic, so we don't care about state. But, we have to keep good maintenance of access list. uRPF is easy to configure, but we need to care about asymmetric routing. The reason we can't stop the IP reflected attack. There is no good implementation of the feasible mode.

PHILIP SMITH: Are there any questions?

Especially following on from Merike has just been talking about.

GAURAB RAJ UPADHAYA: The sorting word, uRPF, you are using a satellite or some other form of transit, it doesn't provide you with asymmetric routes or path, so I know spoofing is bad, but I don't think uRPF is a solution to this. There must be other ways and more innovative ways to do that.

TERRY MANDERSON: Terry from APNIC. I think your efforts should be commended. I would like to see more entities take some steps towards traffic measures. Although, it strikes me that when you consider reflected amplification attacks, all you need to do is look to DNSSec to enable a name server and you have a source. You don't need a recursive name server and you certainly can't turn off DNSSec.

SPEAKER FROM THE FLOOR: Following on from what Terry just said, in the IETF we have been looking at the whole DNS reflection mode, you can mitigate things of the DNS server, but the other name servers, you turn that off, anything out there that is a a P2P-based service, it is not reflective. You can fix it by urning them into TCP-based servers, but ultimately there is no solution to this. Specific technology for ingress filterage affects all this, but I don't think you can get there with ingress filtering, nothing else will work.

PHILIP SMITH: Any other questions or comments?

OK, well, we will close that off. Thanks very much for your presentation.

So, what I would like to do now, it brings us to the end of the APOPS session. So, before you go away, I would like to give you your speaker's gift. Thank you very much. I would like to invite the other speakers, whose names are on the screen to come up and receive their gifts. They have all contributed to the presentations and have come up to give the presentations, so please don't be shy. Do I have to read out your names? Kae Hsu, Shengyong Ding, Sumon Ahmed Sabir, Amante Alvaran, Gaurab Raj Upadhaya, Geoff already got a gift.

OK, so, that brings us basically to the end of the APOPS session.

Now, before I close off a few things I would like to talk about and remind you about. The first thing is if you felt this was a useful format, to have the meeting, please indicate that to the APNIC Secretariat staff at the helpdesk. I don't know if we have feedback from the meeting, if you can indicate the feedback forms. Or if you preferred the format we did at the last time with special interest groups sitting in parallel and you had to run around and find the sessions and you missed sessions because they clashed and so forth, please let us know. If you like it, we will do this again next time.

The second thing I want to mention, we have the lightning talks, which start at 6 o'clock. There is no tea and coffee outside, but we have a short break to refresh. We have five volunteer presentations. They have a maximum of 10 minutes each including set-up time and questions and answers. I will have a bell up here and that is all the time the speakers will get. This is the order that I will be calling them up in (refers to slide). So it will take exactly 50 minutes and not a second longer. Of course, the speakers don't have to speak for this long, but that is all they have. You don't need slides if you don't want slides. If you have lots of slides, well you have 10 minutes. For those of you who are not interested in the lightning talks, which I think you will be missing out, you can go and catch the first bus which leaves at 6:50. You can catch the bus to the social event this evening, which is being held at the British Consulate. The first one is 6:50. I think they return starting from 9:30. The first one is at 6:50 and then 7:00, 7:10 and 7:20.

I would like to say thank you to all the speakers and SIG chairs.

SAVE VOCEA: I was just speaking to Gaurab. He suggests having a short discussion now on whether people liked the APOPS format. And also that people might like to start the lightning talks early, say at 5:45. It is just a suggestion.

PHILIP SMITH: If you like to have a discussion about how it worked, I'm happy. I'm happy to start the talks early, because it gives me 15 minutes more to get ready for the social event and it gives you more time too.

Let's do that. Let's have a short discussion about how the APOPS thing went. Open floor, microphones are here.

GAURAB RAJ UPADHAYA: I like this format. That is what I want to say.

PHILIP SMITH: Anybody else? Can we have a show of hands? Any of those who liked it want to say why they liked it? Just because it is different? What about those who didn't put their hands up? Any reasons why you either don't care or didn't like it? No, just don't care. Just come here to enjoy the atmosphere. Terry?

TERRY MANDERSON: Far be it for me to keep quiet. I really like the format, it allows me to actually be here, listen and participate, as opposed to following the policy structure stuff of APNIC. So it was really good.

IZUMI OKUTANI: This is Izumi from JPNIC. I really liked the session today. I'm not sure if it's to do with the format, but I thought the content was excellent and each presentation was really interesting, so I really liked it.

PHILIP SMITH: I think it was a bit of both, the different structure. The SIG chairs working with the programming committee meant that we could constructively look at some of the content and try and get more operational content. So I think it helped. I think also, people seeing some of the names that have gone along with some of the initial presentation offers made people more willing to contribute also. It has kind been a two-way thing that has helped. Anybody else?

GAURAB RAJ UPADHAYA: What do the chairs think?

PHILIP SMITH: What do the chairs think? I don't know, I just sit up here and introduce presentations, ask questions when nobody else asks questions. I'm at your service. I mean, OK, you asked for an opinion, I actually quite like this format.

If we give a little bit of history, it was Anne Lord and I talked about this a few years go about integrating operational aspect with the APNIC meetings. It is not just having the APNIC meeting talking about addressing policies.

One thing in the Routing SIG is we were getting routing presentations, but also offers of operational presentations that weren't quite to do with routing.

Merike talked about security issues. If we had in the previous format we would have had no real avenue of place to offer that presentation. So that is an example of the difference it has made. We can now focus a bit more on security stuff.

That is what Anne and I were trying to think about a few years back.

And certainly seeing how the RIPE meeting has revised itself, getting away from RIPE working groups and then focusing on grouping meetings, it has worked well. I think it is something that I think could work reasonably well for this meeting also.

It is very much an experiment. The SIG chairs will be talking about this in a few days' time to see if we can carry on with it. It needs community support to carry on with it as well. We have had some fantastic presentations today, I agree, all of them.

It's not going to work again unless we get the same high-quality presentations the next time around. I would like to encourage people to take a look at the content, tell your colleagues and friends who didn't come today, and say, "Look, this is the sort of thing people were talking about." There are costs to get from Bangladesh to Taiwan and time. Sumon recorded his presentation today. Record your presentation, it is accepted by the program committee, record it and we can play it here. It gives one or two more headaches for the technical team at the back, but it is a nice way of being more inclusive of the whole region, rather than just covering those people who can travel to where the APNIC meetings are being held.

We have tried so many experiments today and all of them seem to have worked, from my perspective sitting up here. I hope you all agree with that. I guess the proof is if we try it again and get some enthusiastic support from presenters, we know this has actually worked.

CECIL GOLDSTEIN: I would like to add a word as a newcomer. It is an excellent presentation and it reminded me of a conference environment, in so much as the papers presented were of an extremely high standard. I didn't realise it wasn't always the case. I think it would encourage presenting the papers in the full plenary or full meeting, would certainly encourage people to make presentations because, of course, the exposure is more wide and it gives people to attend, as Philip mentioned, to attend all the sessions without having to worry about scheduling and deciding where they want to go.

From an operational point of view, I certainly feel that I would have felt something lacking if there wasn't this kind of presentation.

It looks at the operations and functions which most of you are concerned with. I would certainly feel it is something that should be tried again. I should continue to work in this way.

PHILIP SMITH: Any other opinions or thoughts at all?

AKINORI MAEMURA: I really like the format of APOPS because I could have a lot of experiences from the various countries. It has quite stimulating to me. Frankly speaking, before we had working SIGs and working meetings in APRICOT, there was no session to share the experiences from the operational technology. So this is really good to do such things. So I really, it is really easy to understand. Now, APOPS is, for my case, the Asia Pacific region version of the JANOG. It is good to see the experience from the Asian countries and I'm looking forward to see more and more experience from those countries within the Asia Pacific region.

PHILIP SMITH: Do people have other suggestions about how to encourage more participation? Should we continue with people recording presentations and sending them here? I feel it is more inclusive. I travel over the whole region and people say, "I wish I could get to an APNIC meeting." It is one way of helping with that. Another question that comes to mind is about the special interest groups. The special interest groups have not ceased to exist.

APRICOT 2007 in Bali in February, the IA SIG will be meeting as part of an APRICOT meeting. We are going through developments so APRICOT will look defer to that in Perth. We have done a revision, but the SIGs will be meeting. Of course, with this meeting, the IPv6 technical SIG is having its own event. The NIR SIG is having its own. It is covering topics that don't fit into this forum. It is to reassure people. The SIGs are here and will remain here. The SIG chairs form the program committee at this stage. I'm sure if somebody would like to volunteer to help, we would be delighted to have your assistance. There never can be too many people helping to pull the program together.

Anyway, any other comments? Thoughts? No. OK. Thank you very much everybody for the day. We are going to take a five-minute break to let you stretch your legs, grab some fresh air and then we will make a start with the lightening talks at 5:45. Ryan, you can be poised and ready to start talking then. Thanks very much.