Network is tanking

[deeperbydesign]

I opened a ticket about this about 5 hours ago, but haven't heard anything. I realize that the unixshell# support people are digging out of new account requests due to a delayed piece of hardware, but the network seems to be under attack or having a serious outage.

Anyone else noticing this? I'm getting decent uploads to my vm, but downloads from it are <30KBps. I've tested the transfer rates from/to several hosts on isolated networks so I'm pretty sure it's the unixshell# network or their datacenter.

[happyfunball]

I've noticed this as well. Performance has suddenly gone from excellent to terrible.

[deeperbydesign]

I'm seeing much better rates now, but I'm keeping my fingers crossed. Thanks Matt if you fixed it.

[happyfunball]

Things have improved a little bit for me, but my downloads are still well below 100kb/sec, and they used to be about 300.

Latency is still high as well.

I hope this isn't how the service will look going into the future.

[deeperbydesign]

It seems that the network is still having problems, any news on this?

[Imdsm]

I'm still awaiting my account.
It looks like unixshell# had an influx of signups and have been overcrowded, eh?

[deeperbydesign]

I'm seeing better transfer rates this morning, maybe something's been fixed or the attackers got bored?

[matta]

It was a hacked unixshell# VM, it is difficult to debug these as we can't just enter a Xen VM to see running processes, etc.

I first attempted to rate limit the VM to 10Mbit/s, but it seemed to be a high packets per second attack and finally had to shut the customer down.

[happyfunball]

I'm not sure if this helps you much, but you are already measuring bandwidth for everyone. Why not write a script to show you the biggest users of bandwidth in the last few hours. That might expose a compromised server faster than any manual process snooping.

[QUOTE=matta]It was a hacked unixshell# VM, it is difficult to debug these as we can't just enter a Xen VM to see running processes, etc.

I first attempted to rate limit the VM to 10Mbit/s, but it seemed to be a high packets per second attack and finally had to shut the customer down.[/QUOTE]

[matta]

We have no problem tracking down the host/VM.. we use RTG to monitor the switch ports and then utils on the host to find the offending IP. The problem is finding out if it's a hacked Apache running udp.pl from /tmp/.owned or if it is someone the VM owner gave an account to, or whether the VM owner themselves was running it.

[Surferdude]

I can confirm this. Getting 15k to ftp.us.debian.org

[matta]

Huh?

Bandwidth hasn't been a problem since yesterday... when it was it was uploading that was slow, not downloads. ftp.us.debian.org is a rotating mirror and one of the mirrors is probably slow.


# wget ftp://ftp.us.debian.org/debian/dists/Debian3.1r0/main/binary-i386/Packages
--16:28:20-- ftp://ftp.us.debian.org/debian/dists/Debian3.1r0/main/binary-i386/Packages
=> `Packages'
Resolving ftp.us.debian.org... 35.9.37.225, 128.101.80.133, 204.152.191.7, ...
Connecting to ftp.us.debian.org[35.9.37.225]:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD /debian/dists/Debian3.1r0/main/binary-i386 ... done.
==> PASV ... done. ==> RETR Packages ... done.
Length: 12,981,063 (unauthoritative)

100%[====================================>] 12,981,063 1.00M/s ETA 00:00

16:28:31 (1.19 MB/s) - `Packages' saved [12,981,063]

[matta]

Again, it uses a different mirror this time.


[root@vm11 ~]# wget ftp://ftp.us.debian.org/debian/dists/Debian3.1r0/main/binary-i386/Packages
--16:31:39-- ftp://ftp.us.debian.org/debian/dists/Debian3.1r0/main/binary-i386/Packages
=> `Packages'
Resolving ftp.us.debian.org... 128.101.80.133, 204.152.191.7, 204.152.191.39, ...
Connecting to ftp.us.debian.org[128.101.80.133]:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD /debian/dists/Debian3.1r0/main/binary-i386 ... done.
==> PASV ... done. ==> RETR Packages ... done.
Length: 12,981,063 (unauthoritative)

100%[====================================>] 12,981,063 3.61M/s ETA 00:00

16:31:44 (3.05 MB/s) - `Packages' saved [12,981,063]

[Imdsm]

The networks seeming fine downloading, its just a little slow uploading.
It might be the fact I havent got a decent plan but it was steady at like 98kb/s yesterday.

[happyfunball]

The network appears to be crawling again. Downloads from my vps aren't breaking 70kB/sec. This is far below the usual 300kB/sec.

Whats happening to the network lately?

My vps at Tektonic is also crawling along, though even worse, at 23kB/sec !

[happyfunball]

Its OK again now. Strange. I'll keep an eye on my end to see if the problem is here.

[xiongy]

This sucks. My VM is unusable (VM5)
Even simple/light pages are taking forever.

[nsuan]

I seem to be having bouts where I can't make an outogoing connection at all


vash:~# telnet google.com 80
Trying 216.239.57.99...




And it just sits there until I cancel it.

[btm]

I haven't been able to keep my ssh sessions open long enough on vm10 to figure out where my issues lie. Connections to my console die after a minute or two also. And I've been getting "invalid domain" and "Cannot connect to console XX on domain XX" errors when logging into the console. I had one vps go down for hours the other day, wouldn't start in teknic, and I got the invalid domain error when trying to connect to the console. Was able to start it the next day. I opened a 911 ticket when it went down, and I haven't had a response yet, it's been about 36 hours. I've opened two tickets since, and have yet to hear any word back from anyone, except a sales guy who assured me that nothing was wrong anywhere. :confused:

If anyone is out there from Unixshell, please look at support tickets 11468, 11589 and 11769.

btm

[jvincent]

Since yesterday I have had many issues using any TCP-based protocol including : SSH, Webmin HTTPS port 10000, SMTP, IMAP, this forum, Control Panel.

I have no problem accessing other hosts using same SSH client.

Ping seems to work properly with little or no jitter and just a few packet lost.

I'm on VM8, using the latest patched debian 2.6 kernel.

With SSH, the symptoms are:
- If I get through, after issuing a few commands, the connection hangs.
- Sometimes it hangs at login or authentication.
- When I finally can issue commands, 'who' shows my other hanged connections.
- Using 'top' does not show anything abnormal (less than 0.10).

I have stopped most processes (except SSH and Webmin) but still have the same issue.

The fews logs I have checked don't show anything abnormal apart from SSH connection attempts failures.

This is very anoying, as I use this as my company's mail server which is down now. Can't ever send emails with attachements since the connection always hangs before the end, only small emails can get through.

I have read a message from Matt mentioning a Firewall issue yesterday, telling us it was no longer used, but it still does not work for me.

I remember that in another data center (not unixshell) I had a similar issue which was due to a faulty Cisco Ethernet Switch that had to be replaced. I know this sounds strange cause a switch is not supposed to act at TCP level but this was the case. Maybe due to maximum packet size issues that TCP uses during file transfers.

So I just tried pinging with larger packet sizes and it shows that the maximum packet that gets a response is 1464 bytes (of payload) long. At 1465 there is no response. Bigger packets at yahoo.com work fine.

I then did same test using the don't fragment option and here's what I got: size 1464 does not require fragmentation while 1465 is the limit where gragmentation is needed and also the limit where unixshell stops responding.

I therefore decided to reduce the MTU for my interface and tried SSH again, but this still does not work.

Conclusion, the problem does not seem to be linked to packet size but really TCP at unixshell, this is weird unless there is some kind of faulty TCP gateway.

Please help,
Jean.

[jvincent]

The problem appears to have been fixed now.

It would be wonderful to know what happened and if there is a long term solution to the problems of the last 2 days.

Jean.

[will]

Thanks for the info Jean! I've been running a ping monitor and noticing occasional blips of packet loss. Also noticed that my jabber connections are dying at random times. I'll keep an eye on it today, hopefully won't happen anymore.

[btm]

[QUOTE=jvincent]The problem appears to have been fixed now.

It would be wonderful to know what happened and if there is a long term solution to the problems of the last 2 days.

Jean.[/QUOTE]

I've had a couple ssh sessions to my servers for about 10 minutes now, which is a record. I've noticed that ICMP is pretty stable, it's just the connection oriented sessions, like there is a middle man device somewhere than drops the ball after a while.

I still haven't heard from anyone about my tickets from Wednesday about this.

btm

[gregthegeek]

Yeah, my ssh (putty) wont stay connected for too long anymore. I at least can have it up for 20-30 minutes, but still, it's annoying when it tanks all the time. I used to have it up for hours without error. What's the deal guys? Obviously it's not me or my VM or my server, it's on your network.

And, message to admins, what's the deal with putting a firewall in front of our servers? I thought we were supposed to have *open* access to these things. Maybe I am mistaken. I know there was some issue with an IPS system. I don't know about anybody else here, but I would, believe it or not, much rather not have my hosting provider firewalling my VPS. I'll setup my own iptables if I want. Plus, if I really want security, I wouldnt be on a shared system. As long as we all take the basic precautions, security is not really a big concern to me. (not that it isnt important, it absolutely is, just not critical for what I host, if you know what I am getting at.)

[matta]

It is still a problem? All firewalling/IPS has been off for a while now.

[gregthegeek]

You know, I had a problem with that for the last day or so, and now, after I posted that, I have had my ssh open for hours. Go figure.

Glad it's working. Thanks for asking.