netstat foreign address, apache2 server

[sri]

Hi,

Yesterday my Apache2 server started running slow.
Same today. So I did a netstat and got this:

==============================================================================
tcp 0 0 [MY SERVER]:57922 gp5s01.manabi.city.:www TIME_WAIT
tcp 0 0 [MY SERVER]:47364 carrera.websitewelc:www TIME_WAIT
tcp 0 0 [MY SERVER]:42965 apache2-twiddle.kea:www TIME_WAIT
tcp 0 0 [MY SERVER]:60629 eo-in-f147.google.c:www TIME_WAIT
tcp 0 0 [MY SERVER]:60663 eo-in-f147.google.c:www TIME_WAIT
tcp 0 0 [MY SERVER]:52982 62.129.142.107:www TIME_WAIT
tcp 0 0 [MY SERVER]:42619 niobe.sweb.ru:www TIME_WAIT
tcp 0 0 [MY SERVER]:43560 w8a.rzone.de:www TIME_WAIT
tcp 0 0 [MY SERVER]:43666 minsk.hostforweb.ne:www TIME_WAIT
==============================================================================


[I'm kinda clueless when it comes to networking, so please bear with me.]
I stopped Apache and all these things went away.


I don't understand how people from remote sites are able to access
my server at all the different ports.

Here is my iptables -L:

==============================================================================
Chain INPUT (policy DROP)
target prot opt source destination
ACCEPT tcp -- anywhere [MY SERVER] tcp dpt:7322
ACCEPT tcp -- anywhere [MY SERVER] tcp dpt:7373
ACCEPT tcp -- anywhere [MY SERVER] tcp dpt:smtp
ACCEPT tcp -- anywhere [MY SERVER] tcp dpt:www
ACCEPT tcp -- anywhere [MY SERVER] tcp dpt:https
ACCEPT tcp -- localhost localhost
ACCEPT icmp -- anywhere anywhere icmp echo-request
ACCEPT icmp -- anywhere anywhere icmp time-exceeded
ACCEPT all -- anywhere [MY SERVER] state RELATED,ESTABLISHED

Chain FORWARD (policy DROP)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
==============================================================================


I thought the last rule in INPUT was the problem:
ACCEPT all -- anywhere [MY SERVER] state RELATED,ESTABLISHED

(I made up the rules about 18 months ago when I first got the server and
haven't bothered about it since.)

So I deleted the rule and some strange things started happening:
running iptables -L -- it almost hung (took a long time to finish)
and so did netstat.

Whenever I startup apache all these foreign sites seem to be
accessing mine.

I google for keywords, but did come up with anything.

Can anyone help with what's going on?

[lex]

Okay, a few things. First of all, TCP and UDP connections are designated by four parts: the source and destination host (you knew that), the destination port (ie 80, for HTTP), and the source port. Usually the source port is invisible to you, the user. Next, "TIME_WAIT" is normal: it's a state that a connection goes in after it's been closed. TCP enforces a certain amount of waiting before another connection from a given host/source port can be made to a given host/destination port. Usually you'll see connections sit in TIME_WAIT for 5-10 minutes.

Now, this line:

tcp 0 0 [MY SERVER]:57922 gp5s01.manabi.city.:www TIME_WAIT

That means that a connection was opened FROM your server TO gp5s01.manabi.city.whatever's HTTP server (www is port 80, check /etc/services for the mapping). The connection came FROM your server's port 57922. Did you connect to those web servers for some reason rather recently? If not, start wondering about whether you got hacked.

Now the iptables stuff:

Chain INPUT (policy DROP)
target prot opt source destination
ACCEPT tcp -- anywhere [MY SERVER] tcp dpt:7322
ACCEPT tcp -- anywhere [MY SERVER] tcp dpt:7373
ACCEPT tcp -- anywhere [MY SERVER] tcp dpt:smtp
ACCEPT tcp -- anywhere [MY SERVER] tcp dpt:www
ACCEPT tcp -- anywhere [MY SERVER] tcp dpt:https
ACCEPT tcp -- localhost localhost
ACCEPT icmp -- anywhere anywhere icmp echo-request
ACCEPT icmp -- anywhere anywhere icmp time-exceeded
ACCEPT all -- anywhere [MY SERVER] state RELATED,ESTABLISHED

You can look this up in the iptables manpage, but I'll give a brief summary of what this does: It says that by default, packets arriving from the outside world at your server are DROPped. If they're going to port tcp 7322, 7373, 25 (smtp), 80 (www), or 443 (https), they're ACCEPTed. Traffic from your server to itself is accepted. Pings (echo-request) are accepted. Ping responses (I think that's time-exceeded) are accepted. Then that last rule, the one that's twigging you: that says that any traffic RELATED to a connection you ESTABLISHED is allowed, ie, response to any kind of connection you originated.

So, if you recognize those servers in the first section and you know you made those connections, then everything else looks good. If those connections up top don't look familiar, maybe you've been owned.

[sri]

Thanks for the response, lex.

I don't recognize any of the website (unless I am totally forgetting something, I don't even know why I would be connecting to any of those sites -- I just run few services from my machine...)


The reason I got confused was because whenever I stopped Apache those connections (as you say, from my server to remote servers) go away (from the netstat report). As soon as I start Apache, they come back. Why would this happen?



But any pointers to docs on how to proceed from here? Those connections still keep coming back when I start Apache. Do I change root and all user passwords and reload a new kernel and system from scratch?



(Also, can the Xen be causing this problem? That I somehow see other people's Xen instances sitting on the same machine -- all their connection can be see by netstat...?)

[sri]

I think this is the problem:

Someone on the same "subnet" (sorry, grasping for terms here) is accessing remote servers through my Apache server:

I did "netstat -an"
and the the local address it showed wasn't mine but
matched mine closely:

My IP: xx.yy.zz.AB
Other IP: xx.yy.zz.CD

Also my policy on OUTPUT is ACCEPT by default.

So my guess is someone on the same subnet as me is accessing because of this Iptables rule. To make things secure, all I have to do is either block the IP addr that isn't mine or switch the OUPUT's default accept policy to DROP and add only my IP with an ACCEPT policy....


Does this all make sense?

-------------------------
Update
OK despite the above (even after setting iptables as mentioned above), the other person's IP address still accesses remote sites when I start up apache...
the local address shows up as:
tcp6 0 0 ::ffff:[OTHER_IP] ::ffff:[SOME_REMOTE_IP] TIME_WAIT


-----------------------------
Apaches modules installed (in case it helps):

aalias
auth_basic
authn_file
authz_default
authz_groupfile
authz_host
authz_user
autoindex
cgi
dav
dav_fs
dav_svn
dir
env
fastcgi
headers
info
mime
negotiation
php5
rewrite
setenvif
ssl
status
userdir