Whirlpool

My website hosted at Web24.com.au (in Fujitsu data centre in Melbourne) is currently having packet lost issue. mtr shows around 20% packet lost at fujitsu's router. Now my website will render the heading + some content, and then browser just stalls there trying to download the rest of the page...

Worse, the site Web24.com.au is inside fujitsu's data centre as well, which means I can't connect to it to get the support details, nor connecting to HSPc to submit support tickets.

Anyone else at Web24 that is having similar problem?

Just received calls from three clients who have sites hosted by companies located in the Fujitsu building, same problem.

No one wants to answer their phones, either.

Not happy.

I hope we don't get smacked for this...but here goes.

We also have servers in the Fujitsu datacentre along with Web24.

We have been in touch with the tech's down there that confirm that it is a faulty router from Verizon.

Verizon technicians have been called and have responded stating they are currently working on the situation, however they could not provide us with an ETA at this time.

I can post an update once I hear something back (if mods are ok with that).

Regards,

Dale Evans
Manager

I know of some mission critical sites in fujitsu that are running fine.

So either the issue is now fixed, or they have a separate network (with Verizon also) for the critical stuff.

EDIT: Ignore me, I was checking Fujitsu sydney...

Ahh glad I know whats going on....I noticed something odd going on when I was checking email after 11.

Got a reply back from Web24 and it seems to be an issue with Fujitsu's upstream provider. They have now failed over to the secondary link (it wasn't so automatically because the primary link wasn't really "down" in the first place).

Has anyone noticed the same sort of issue popping up again?

They must be trying some things again.

yes. my server at web24 is having VERY intermittent access.

it was down the other night too.

It just came back up.

I noticed it was down for 10 - 15minutes.

I wonder what was going on this time.

Sigh it just dropped out again, hope they get it sorted soon.

Just noticed that some funny stuff is happening. It seems I can't get to one of my servers in the US from home which is on a telstra bigpond adsl service but if I try to get to the same server from a server in one of our offices in hobart which is on a internode dsl account I have no problems. Able to access other servers in the US without problems and have no issues with accessing the server from there.

I am noticing though that traceroutes coming back to australia are failing to a number of services I have tried:

$ traceroute abc.net.au
traceroute to abc.net.au (202.6.74.117), 30 hops max, 40 byte packets
1 72.249.3.233 (72.249.3.233) 0.579 ms 0.567 ms 0.550 ms
2 switch20.rimuhosting.com (65.99.195.131) 0.480 ms 0.429 ms 0.630 ms
3 206.123.64.37 (206.123.64.37) 0.467 ms 1.551 ms 0.477 ms
4 border4.g3-2.colo4dallas-3.ext1.da­ l.pnap.net (216.52.189.9) 0.591 ms 1.688 ms 2.102 ms
5 core1.tge5-1-bbnet1.ext1.dal.pnap.­ net (216.52.191.38) 1.682 ms core1.tge5-2-bbnet2.ext1.dal.pnap.­ net (216.52.191.103) 1.516 ms 2.185 ms
6 sl-st20-dal-8-2-1.sprintlink.net (144.228.250.113) 1.665 ms 2.516 ms 1.013 ms
7 sl-bb27-fw-5-0.sprintlink.net (144.232.9.137) 2.148 ms 3.035 ms 2.376 ms
8 sl-crs2-fw-0-12-0-0.sprintlink.net­ (144.232.11.134) 3.735 ms 2.975 ms 3.804 ms
9 sl-crs2-ana-0-8-0-2.sprintlink.net­ (144.232.18.52) 31.078 ms 31.102 ms 101.583 ms
10 sl-bb21-ana-15-0.sprintlink.net (144.232.1.60) 32.883 ms 31.187 ms 32.519 ms
11 sl-st20-la-13-0-0.sprintlink.net (144.232.20.67) 32.221 ms 32.204 ms 32.232 ms
12 sl-newzeal-137399-0.sprintlink.net­ (144.232.154.242) 32.494 ms 38.003 ms 40.008 ms
13 so0-2-0.labr5.global-gateway.net.n­ z (203.96.120.101) 32.346 ms 32.586 ms 52.015 ms
14 so1-1-2.sybr5.global-gateway.net.n­ z (202.50.232.74) 181.245 ms * so-1-3-2.0-sjbr2.global-gateway.ne­ t.nz (203.96.120.118) 180.189 ms
15 so0-0-0.sybr4.global-gateway.net.n­ z (203.96.120.86) 186.495 ms 181.074 ms 181.210 ms
16 connect1.sybr4.global-gateway.net.­ nz (203.96.120.122) 186.953 ms 181.596 ms 181.752 ms
17 DABC141265-1.gw.connect.com.au (210.8.2.249) 264.162 ms 181.740 ms 182.734 ms
18 * * *
19 * * *
20 * *

Hi. For full information on the issue as it unfolded please see: support.web24.com.au/ind...12&group=default

Web24 Management met with Fujistsu Nataional Datacenter manager, National Networks Manager and Melbourne DC manager yesterday afternoon to discuss this issue. Together we have put in place an escalation procedure to fail over to the secondary link immediately should a non-hard failure occur (packets still able to traverse link) on the primary link

We offer our sincere apologies to our customer for this service interruption

Mosky writes...

No one wants to answer their phones, either.

HI Mosky.

Was this quote in regard to Web24? We had 2 on call technicians who were fielding calls during the incident and were able to communicate with our customers effectively.

Our 1300 number, option 5 is for emergency support.

What exactly happened today Aaron?

Yesterday was explained in detail but nothing has been said regarding what occurred today.

At this point it appears to be a continuation of the failed main link compounded with our HSRP and Failover firewalls flapping.

Please note that we post information on our website, not on whirlpool forums, and this should be where you look for further service information and updates.

Please feel free to contact us directly should we be able to assist further.

A. Sinclair writes...

HI Mosky.

Was this quote in regard to Web24?

Nay, sorry I should have been clearer. It was in relation to other providers in the Fujitsu building who are selling hosting.

Nothing to do with Web24. :}

A. Sinclair writes...

Please note that we post information on our website, not on whirlpool forums, and this should be where you look for further service information and updates.


Just to let you know, at the point where I posted the question about what happened...there was no updates on what occurred today so going to the website would have yielded no result.

Edit: There is still nothing on the website tonight about what happened today.

Is it all on the blink again?

I tried to call Web24 Emergency Support but 1 tech was on the phone each time I called, 1 had the mobile off and the 3rd number it goes to gives a denied message.

Yeah, same here. i think it is indeed on the blink sir! Also no luck getting through to the on call tech. i think it went down around 8:30pm tonight. Fingers crossed they know about it, and fix it shortly.

from what i can see, seems to be major packet lose...

Its always when you want to update something on the website isn't it grrr!!

Yep, major issues here. All my sites are down and even web24's own website is cactus. I simply can't connect to any of them via http or SSH or anything else.

Quite frustrating, hope it gets sorted out soon.

Yup, i called the tech at 8:30pm with the "we have tech's looking into it now" response.

The site seems to get some pings through, so my bets are on a faulty router somewhere in the mix.

Fingers crossed...

Crikey, I just bought hosting off them a few days ago and was in the middle of building a site!

We've got servers in the same DC (Fujitsu) and we too are currently experiencing issues.

Once I get an update, I will post up for all.

Regards,

Dale Evans
manager

Thanks Dale!

I have one of my sites with them but i have web24 hosting through a reseller of them. My site is down and i have done a traceroute which doesn't look good. After hop 11 or 12 there are a lot of timeouts.

Hi all,

We also have servers in Fujitu's DC. I've been speaking to colleagues and they're in contact with Web24.

Aaron is unreachable as he is currently overseas for Parallels Summit.

Ok, I have confirmation from techs in the DC.

The cause is a major DDOS attack from Hungary causing router and traffic issues.

I've been told that the IP has now been blocked, the issue isolated and routers are being worked on now to bring all services back to normal in a very short time.

Not exactly an answer we all wanted to hear, but its an answer none the less.

I am getting phone calls as info comes to light, so I will pass the info on asap.

Dale

Just got another call...

Apparently everything should be restored within 30 mins.

Those still having issues after that time should contact their relevant providers by the appropriate methods.

Those who are clients of Hightek Hosting are welcome to contact us by phone on 02 5908 2370

Regards,

Dale

Thanks Dale, much appreciated.

Just an update.

Connectivity has been restored to equipment at Web24. Hopefully there won't be any more downtime. *touches wood*

Yup, my server is back up too.
Now to plot against Hungary... :)

Thanks Dale and Gav,

Service seems to be running fine now!

Thanks for the updates, much appreciated.

Hi everyone,

I can access my server again. However, outgoing connections from the server (e.g. a webservice connection) to another server are slow.

When I do for example

wget forums.whirlpool.net.au

on my linux web24 vserver it takes about 30 seconds until the site is loaded.

From my local machine it takes only 1 second. Anybody else having similar issues with outgoing connections?

Cheers,
Bjorn

Hi Bjorn,

I've just logged into one of our servers that is in Web24 and while outgoing pings seem ok;
Pinging jgk-cs.com.au [116.240.200.31] with 32 bytes of data:

Reply from 116.240.200.31: bytes=32 time=4ms TTL=120
Reply from 116.240.200.31: bytes=32 time=4ms TTL=120
Reply from 116.240.200.31: bytes=32 time=5ms TTL=120
Reply from 116.240.200.31: bytes=32 time=4ms TTL=120

Web traffic doesn't seem to want to load quickly at all. It could be a side affect of something that hasn't updated properly after traffic has been re-routed.

I just logged in and outgoing http traffic seems to be working properly for me again.

Cheers,
Bjorn

Before I throw my 2c into the mix here, just making clear I'm not having a shot at either the hosts who use Fujitsu or Internet products sold by datacenters in general however something doesn't sit right here.

As a datacenter selling transit we can assume they have a fairly substantial amount of capacity. We have three hosts here, who, with all due respect, certainly would not be pulling data in the volumes some of Fujitsu's larger clients are. This must have been a rather large DoS attack to degrade performance so severely. We are not talking about your typical colocation provider here with 6 or 7 racks and something like 30mbit of capacity.

Also, for what its worth, simply blocking the source IP of an attack will not stop the traffic from consuming as much of your link as possible. The most effective way is to simply drop the destination prefixes, our announce them to a smaller, unused, less expensive route.

From what I heard last night, the attack not only affected Web24's network, but a lot of others who use the Verizon network, with Web24 being the worst.

Unfortunately with this type of attack, dropping the link will stop the attacks for a small period of time before the attack then re-routes down the new route which has been created and kill that as well.

Not many networks in Australia have a really good way to divert DOS attacks mainly because of the price of bandwidth here, so that being the case the only way to stop it completely is to get to the source of the attack and take it out of the equation.

Everything seemed to be up and stable at around 3am this morning when I went to bed after some of the default routes were changed to one of the other providers to minimize the effect of the attack that was still occuring.

why they attacking the network?

Had problems all week, and intermittently over the last month. Had some SQL scripts that seemed to be taking randomly long times to run too. Not sure whether to stick with them. They have been very responsive on the few tickets we've submitted, but I've become less convinced by the 99.9999 uptime claim on the home page. Anyone else seen anything?

mike_something - it may seem like the best thing to do as soon as a host sees some downtime. As you experienced firsthand, the support has been great from them....support and service is #1.

Also, just check what 99.9999% uptime actually means, I'm sure it's in the fine print somewhere.

I would stick with them as I'm sure the problems would only be very temporary.

I've been with them for a few months and can honestly say they've been fairly well rock solid except for the other night.

I have a server with another provider and let's just say it has been less than trouble free....

Joel R. writes...

Also, just check what 99.9999% uptime actually means, I'm sure it's in the fine print somewhere.

Quite out in the open actually.
www.web24.com.au/web24/2...l_agreement.html

Does that mean we're eligible for a 10% credit? It was certainly down for more than 43 minutes :)

(Edit... it may well come under condition 5a though, as a DOS attack is pretty much out of their control)...

i have been looking at all hosts' uptime guarantee and with all the exceptions, basically the guarantee means absolutely nothing. noone has the guts to stand up for their own service quality and guarantees.

id ike to now what are the inclusions, as its seems there is nothing left in the equation after all the exclusions are added.

I submitted a ticket to web24 asking for for some kind of (remotely hosted) service blog, so when something happens, I can actually find out. They responded with some kind of "will be submitted to management " response. I figure that if anyone else wants it they should submit a ticket too. Or maybe someone else has got a better idea?

web24 appears to be down again.....

Yes, went down 10 mins or so ago....they know about it, another DataCentre issue.

Yep, down yet again. I have had two sites with these clowns for about 2 months now and it's gone down several times already. Time to move away from these morons.

And it appears to be back to normal!

Performing trace to web24.com.au...

1 FastEth8-1-0.sb1.optus.net.au (192.168.34.2) 2.868 ms 2.651 ms 2.308 ms
2 61.88.221.1 (61.88.221.1) 2.947 ms 2.631 ms 2.452 ms
3 202.139.39.250 (202.139.39.250) 2.940 ms 2.754 ms 2.842 ms
4 0.ge-6-2-0.XT4.SYD2.Alter.Net (210.80.34.53) 2.773 ms 2.741 ms 2.653 ms
5 0.so-0-2-0.XL2.MEL1.Alter.Net (210.80.33.218) 21.404 ms 16.167 ms 15.941 ms
6 210.80.34.222 (210.80.34.222) 17.222 ms 17.270 ms 17.136 ms
7 fuji865-mel-gw.aspac.customer.alter.net (203.166.43.166) 19.334 ms 19.191 ms 19.040 ms
8 * * *
9 * * *
10 * * *

traceroute to web24.com.au (203.16.60.12), 30 hops max, 40 byte packets
1 vlan250.lon-service6.Melbourne.telstra.net (203.50.2.177) 0.31 ms 0.202 ms 0.314 ms
2 TenGigabitEthernet9-1.lon42.Melbourne.telstra.net (203.50.80.52) 0.407 ms 0.256 ms 0.325 ms
3 ozemail12.lnk.telstra.net (139.130.248.10) 0.412 ms 0.414 ms 0.473 ms
4 32.so-2-0-0.XL1.MEL1.ALTER.NET (210.80.32.13) 0.42 ms 0.39 ms 0.47 ms
5 210.80.34.206 (210.80.34.206) 0.413 ms 0.428 ms 0.467 ms
6 fuji865-mel-gw.aspac.customer.alter.net (203.166.43.166) 4.05 ms 5.69 ms 3.395 ms
7 216.14.205.3 (216.14.205.3) 4.649 ms 5.848 ms 6.015 ms
8 * * *
9 * * *
10 * * *

I dont think its a DC issue more like a Host issue.

come on a DC having this much downtime would be a joke...

maybe it time you guys have some serious talk with your host cause i smell BS!!

Max1007 writes...

maybe it time you guys have some serious talk with your host cause i smell BS!!

It isn't the host, it is the DC as another provider who uses the same centre has the same issues!

I am jumping ship as soon as I find an option that suits, this is like the 3rd or so issue this year.

Genthoon I bet the other host is hosted with web24 also...

whim me the host name

Genthoon writes...

I am jumping ship as soon as I find an option that suits, this is like the 3rd or so issue this year.

So what you are saying is that you want a hosting service that goes without ANY issues at all? You want 100% uptime, all the time?

It's simply not possible and anyone who's been in the game long enough knows this. No matter what any "fly-by-nighter" tells you, 100% uptime is impossible!

I would hardly consider 3 or 4 issues to be anything from normal. Like any electronic piece of equipment and the internet...stuff fails/breaks...and it's to be expected.

Last time I checked, the only thing that was even somewhat close to being 100% foolproof was the string between 2 cans...providing the string didn't break of course!

That said, we've been running gear out of the Fujitsu DC for some time now and sure, there have been bumps in the road...but this is technology and it breaks, like anything else.

That said, if you have an issue with a provider out of Fujitsu DC, why not discuss this with your provider?

Regards,

Dale E

Dale E writes...

So what you are saying is that you want a hosting service that goes without ANY issues at all? You want 100% uptime, all the time?

Come and attack me on whirlpool why don't you Dale!

I do have to wonder where exactly in my very short and brief statement, I said I expect 100% uptime?

No I do not look for 100% uptime, but I have with experience (I haven't been doing this for 5-10 mins, more like years) been with providers that have been able to offer me service with minimal interruption (1-2 breif outages a year).

It's simply not possible and anyone who's been in the game long enough knows this. No matter what any "fly-by-nighter" tells you, 100% uptime is impossible!

As I have said before I did not say anything about 100% uptime, and I have been in this long enough to know its not possible.

I would hardly consider 3 or 4 issues to be anything from normal. Like any electronic piece of equipment and the internet...stuff fails/breaks...and it's to be expected.

Fujitsu has had this "bumpy road" as you say several times this year, and at least one occasion at the end of last year. Several of the previous occasions were not as minor as tonight as well, stretching anywhere from 1hr to 4 or more hours easily.

As I have previously said I have been with other providers and not experienced several issues like this in such a short period (ie: 3-6 months).

That said, if you have an issue with a provider out of Fujitsu DC, why not discuss this with your provider?

Whirlpool is a discussion forum and I merely stated 1 line. It did not go into detail at all and it was a possible thought, not a plan of action.

Perhaps this outage has got your stress levels up or something.....go chill out for a while or work on something else and relax a bit...its simply a discussion.

Genthoon writes...

Come and attack me on whirlpool why don't you Dale!

Attack? It was not an attack. If I was going to attack you...well, fact of the matter is, I wouldn't. It's not my type nor is it my standards.

I do have to wonder where exactly in my very short and brief statement, I said I expect 100% uptime?

No, you didn't state it directly, however it was seen as implied. I consider 3-4 outages to still be well within the realms of acceptable in any provider (within reason of the nature of the outages)

No I do not look for 100% uptime, but I have with experience (I haven't been doing this for 5-10 mins, more like years) been with providers that have been able to offer me service with minimal interruption (1-2 breif outages a year).

Well then that opens up the question as to why you are not with said providers now if you were so happy with their service? But there is no need to answer that.

Despite all that was said, I am not prepared to drag this off topic and start a debate on who said what, where and why.

If you wish to take this up with me externally, then feel free to do so, but I don't believe it will help.

End of story.

Dale E writes...

Well then that opens up the question as to why you are not with said providers now if you were so happy with their service? But there is no need to answer that.

Of course there is reason to answer that when you attempt to make me look bad on a public forum.

Implying things is wrong as you are never always right in doing so.

We aren't all rich and we don't all heavily rely on our hosting to be the breadwinner so the choice in movement was purely based on cost and secondly on service.
As with many individuals, I check a selected provider out and experiences have been good apart from the outages.

Despite all that was said, I am not prepared to drag this off topic and start a debate on who said what, where and why.

You dragged it off as soon as you had to make a comment on my statements.

Make yourself look more professional in the future and do not involve yourself in such discussions.

Genthoon writes...

Make yourself look more professional in the future and do not involve yourself in such discussions.

If I want/need your advise on how to run a business, including my own, I'll be sure to let you know ;)

...but we seem to be doing pretty well on our own, but thanks anyway :)

In other news...

Fujitsu issues should now all be resolved and from the brief report I received from the technical team, it was an internal operations issue which caused the outage.

In light of this, no further information has come to light yet from Fujitsu, but hopefully we get something more by morning.

meh.. take it outside boys.. Its Hosting.. if u aren't happy why don't u just change like everyone else does. Just give u a tip for future reference.. the grass isn't always greener on the other side. A bit of pack loss never hurt anyone :)

We have been having serious troubles where clients cannot connect to any websites on our server (in the fujitsu datacentre) but people on different networks can. I have asked Web24 and they cannot find the problem :|
Regards
Cameron

Well. As the thread OP from 4+ months ago, I have long moved on. I agree that you cannot demand 100% uptime, but the amount of network-related issue at Fujitsu just leaves more to be desired...

Unfortunately every major outage experienced over the last 11 months has been directly attributed to the Data Center providers contracted by Web24. We agree that this is an unacceptable situation and have exhausted all means of working with the existing DC providers to a resolution.

From a systems perspective we have had no serious outages within our control and have provided exemplary uptimes. Our many resellers will attest to our systems and support provided.

Web24 started the process of acquiring new data providers aprox. 5 weeks ago and is working towards a situation independent of the DC providers.

Rest assured that we are EXTREMELY FRUSTRATED with this situation and are working to a suitable resolution in a timely manner.

We welcome any questions you may have regarding this situation and are happy to answer them in person.

Down yet again!

Yep, down for me as well.

yes... down... this is extremely frustrating...

regardless of who is at fault, its hard to stay loyal when your sites are down for extended periods of time, with little to no customer advice...

oh well, back to the drawing board

Back up!

Yep, luckily i moved my sites yesterday...