Whirlpool

I have had a very bad experiance with Amnet Support over the past.

A couple of months back I had a guy from russia UDP flooding my address with packets. I captured this with a tcpdump and forwarded my findings to Amnet by lodging a support case. Their responce was we cannot help you, we cannot block addresses on our core router (something I'd think an ISP should do). This was using over 20GB per day of my ADSL usage. Thankfully they stopped spamming me after a while 60GB later in quota usage. I configured my router at my end to drop the packets and not reply, however this does not stop them coming down my ADSL line.

I now have another case with amnet where routes to sites like myspace.com etc are taking over 30 hops to get to their destination resulting in pages taking over 45 seconds to load. Again the responce after multiple emails and support calls is "sorry we cant help you". I said to the guy on the phone, look if you can't help me, I need to change provider. His responce was "fine". I represent 4Play Games Internet Cafe. We have 3x ADSL lines into our store, 2 with amnet and 1 with iinet for redundancy encase Amnet have issues. We are also opening another store in Dunsborough this month, and were also planning to go with Amnet for the provider. Our customers are complaining about the internet being extremely as a result i'm currently routing all data out of the iiNet ADSL2 connection.

Has anyone else had issues like this with Amnet? Is there any other providers out there that provide a higher level of support and actually care about their customers?

Any Amnet people reading this, case number CAS-233343-N1SH. Not happy Jan!

You're running a business on ADSL lines? A business who's entire model is based on access to the internet?

I don't care *how* much redundancy you've got. That's a stupid idea. What if the exchange goes down? Both iiNet and Amnet will be out. What if something destroys your MDF? What if someone takes out all four cables with a backhoe?

You're essentially relying on a connection type known to be unreliable, but is used anyway, mostly because it's cheap.

Also, I don't see how the Russia flood thing is in any way shape or form, related to the Myspace thing.

Also, just out of pure interest, because it sounds like some bizarre routing glitch probably caused by you getting your IPs blacklisted on a router somewhere (you run a net cafe, no way in hell you know exactly what everyone's doing with your connection...) Does it happen against all four links?

It's a right pain the backside when someone is attacking you in this way. Even if you are dropping packets at your router you are still paying for the traffic hitting your router. I have seen extreme cases where this amounts to over half a company's data charges.

I understand why ISP's don't allow changing of static's WAN IP's very easily. Often there are the reason that attacks start in the first place is due to lax security, but not in every case. If ISP's swap out an IP they will need to give the old one to another customer and I am sure they don't want attacks from the word go.

I purchase an extra IP range and use this instead of the WAN IP. This way you can juggle your address's around to your hearts content. It put's you in control of the situation, but it does cost a little extra. I guess you get what you pay for.

Managed solutions aside the only ISP that I am aware of that allows you to block traffic at the providers access routers is Telstra's Internet Direct, via their Custdata interface. It's a very handy feature for this type of problem. Again TID connections cost, but you get what you pay for.

I would be interested to know if other ISP's allow blocking at their access routers.

No doubt an Amnet rep would be able to confirm this, but one reason why they would not block traffic on the border routers (you wouldn't block on the core) is that the ISP still has to pay for the traffic on their uplinks. This would be much the same as you blocking the traffic on your router – the traffic still has to come down your link.

I guess for any blocking to be effective, the ISP would need to have the traffic blocked by their upstream provider. I doubt that said upstream provider would selectively block the source IPs going to your IP, they would probably just block your IP – ie no internet at alll until they unblock you.

Hello Kryptor,

In regards to your queries raised.

The DDOS UDP flood that took place was a distributed attack, ie not from a single source, and more then likely faked from where the source is coming from. The way these type of floods are dealt with is that we place a 24 hour block on the Amnet border routers to the destination address of the attack, ie the IP address of your DSL link will be blocked at the border for up to 24 hours. If the attack still continues we will change the IP address on the targeted account, in all cases we have done this the attack simply moves to attack the new destination IP address assigned.

DDOS UDP floods generally do not just start happening. In all cases this has occurred the cause of the attack was due to an end user causing problems with other users, resulting in the UDP flood occuring. Since you run a Internet Cafe one of your users may have caused this to occur.

Also in regards to how the internet routes. On residential based plans we route data differently then our corporate based fibre plans. Corporate clients can request that a route take a different path if they think this is causing problems, we do not allow residential customers to request this. But in extreme cases we can modify the path if it is causing major packet loss issues, which is not the case with myspace.com . Also if you traceroute www.myspace.com you will notice both address's take different paths out of the Amnet network, I have rectified the address space being used for myspace.com (63.0.0.0/12) to use an alternative outgoing path, which may fix the issue that is occurring.

I do not recommended you using residential based ADSL accounts to run a Internet Cafe. We have quite a number of Internet Cafe's based in Perth and the metro area's running through Amnet using Amcom Ethernet based services, all backed with business grade SLA's, which residential services do not have, but it is your choice what type of service you use to supply your business it most critical service.

When logging a fault with our customer support team, if you feel you are not being supported correctly please request for your case to be raised to the level 2 support team, or the customer support team manager. This team has direct access to raise cases as required to network engineers if a serious fault is found.

Amnet take's it customer support extremely seriously and all customer complaint's are dealt with accordingly if raise through the correct channels.

Regards,

- Matthew Murray

-- responce to lightguard --

umm, we cant get fibre in scarborough. ADSL is the only service available and buisness grade DSL does not give us the down speed we need. It can only provide 2mbps down which is not enough.

The linux routers are very good, running roaring penguin. If one line drops out, it will route all date out the other and continue to reauthenticate in an infinate loop until provider comes back up. If provider is down you just get pado packet timeouts in the log.. which does happen.

amnet is on telstras equipment, iinet have their own.

yes it happens for all amnet links, and to multiple sites. hotmail.com is another one, massive delays when clicking on links, up to 30 second delays due to a bad route. iinet is fine.

Trying to decipher what you're saying *blink*

For starters, I'd suggest putting a delay for your re-authentication, because Telstra's BRAS's get kinda shitty with you hammering them and start ignoring you, hence the PADO timeouts.

I'm not seeing any 'bad routes' as you claim and I'm on Amnet (Telstra Wholesale).

One thing to note, is that 'hop count' doesn't actually mean that the link is 'bad', infact with advent of MPLS once the packets enter the MPLS cloud, their TTL isn't incremented, so it 'appears' they only take a few hops to get out of somewhere. (Optus for example do this, unless you think optus run a cable all the way from LAX to Septimus Roe Square? :P). What you need to be looking for is really high latency and/or packet loss.

One problem I'd suspect you're having, is that you're saturating your upstream so your ACKs can't get out fast enough, so your transfers start scaling back.

Got any graphs of your link utilization or indeed any proof that could help Amnet pinpoint the problem if there is one?

-- responce to Matthew Murray --

Thankyou for your time responding to my frustrations Matthew. I would love to upgrade the connection at 4PlayGames to something higher then ADSL, however last time I checked there was nothing else available in Scarborough, so we have multiple ADSL lines, two from amnet and 1 from iinet. Our routers then loadbalance between the connections.

Hello Kryptor,

We supply quite a number of business based clients fibre based connectivity in the Scarborough based area. Business grade DSL is only available off Amnet enabled based exchanges. In this case the Scarborough exchange is not Amnet enabled, and as a result is not available in the area.

Amnet can only offer Telstra based wholesale ports in Scarborough at this stage, so the highest plan available would be 8mbit/384kbit based plan.

I doubt a bad route would be causing up to a 30 second delay. We have had no other clients log faults in regards to the issues you are experiencing on the network, or delays in accessing sites such as hotmail.com, myspace.com etc.

I will get your case number raised to a level 2 customer support representative.

If they are able to replicate the issue you have occurring the fault will be raised to the next level of support accordingly. Level 2 customer support personal have access to test account's on Telstra Wholesale based ADSL service, and also Amnet enabled ADSL, and SHDSL based connections.

Regards,

- Matthew Murray

That would be great Matthew. Would I be able to contact you regarding prices on business grade fibre in scarborough. We are based on Scarborough Beach Road right next to observation city.

"I doubt a bad route would be causing up to a 30 second delay." True, but if one of them links on that bad route had noise etc, it would cause performance issues. I'm a senior engineer working for a mining company, and I spent a while diagnosing what was going on. It appears to be data heading towards certain providers in the USA. I emailed the trace routes through with that case number.

Again thanks for your time.

This morning I tested my amnet connections, data is now routing back through its normal route to sites such as myspace:

Tracing route to myspace.com [63.135.80.48]
over a maximum of 30 hops:

1 14 ms <1 ms <1 ms 192.168.20.2
2 26 ms 26 ms 26 ms lo0.rba-7206-a.wa.amnet.net.au [203.161.95.201]
3 25 ms 26 ms 25 ms 192.168.4.205
4 26 ms 26 ms 26 ms ge0-1.br01.wa.amnet.net.au [203.161.64.5]
5 26 ms 26 ms 27 ms vlan462.o6ss.optus.net.au [59.154.14.113]
6 251 ms 251 ms 250 ms ge-1-0-0-0.laxow-dr1.ix.singtel.com [203.208.149.250]
7 253 ms 253 ms 253 ms 203.208.169.38
8 253 ms 254 ms 253 ms vl341.cs1.lax1.myspace.com [204.16.35.133]
9 247 ms 247 ms 248 ms vl343.cs2.lax1.myspace.com [204.16.35.142]
10 398 ms 399 ms 400 ms 204.16.35.38
11 306 ms 346 ms 262 ms 204.16.35.41
12 261 ms 260 ms 261 ms 204.16.35.57
13 257 ms 257 ms 262 ms 63.135.80.48

Trace complete.

It is no longer routing through NZ like it was for the past three days. :)

Thankyou for your help Matthew.

yer i reckon fibre is the way to go...cb

Kryptor writes...

It is no longer routing through NZ like it was for the past three days. :)

-blink-
you realize it's still actually going via the same optical path to the US, right (and yes, this includes New Zealand and Hawaii too!)

As I mentioned previously, those hops are still there, optus just hides them within their MPLS cloud.

Hello Shorty,

Hope things are going well?

You are correct that MPLS hides alot of the hops. Trying to explain this to an end user who simply ignores the fact the ICMP response time is 100ms less when routing via NZ in some cases, they still believe that since the hop count is larger the shorter hop path must be better. Of course this is not always the case :(

Regards,

- Matthew Murray

Shorty writes...

you realize it's still actually going via the same optical path to the US, right (and yes, this includes New Zealand and Hawaii too!)

By any chance do you mean Southern Cross Network?
Here is a nice graphical explanation as to how it passes New Zealand and Hawaii. http://www.southerncrosscables.com/public/Network/default.cfm

Unless they make the packets go to Singapore.