I worked 17.5 hours yesterday. Took a half an hour break for dinner between jobs, and that was about it for rest. In Ann Arbor, most of the client machines could not connect to anything but the file server. Knowing that the file server uses Appletalk as the protocol and TCP/IP is used for everything else, we narrowed down the list of possible problems. Checking out the error logs on our servers showed stuff like this over and over and over:
Aug 16 18:54:45 beer.pub.umich.edu dhcpd: DHCPDISCOVER from 00:50:e4:30:6f:24 via eth1
Aug 16 18:54:45 beer.pub.umich.edu dhcpd: DHCPOFFER on 141.211.148.76 to 00:50:e4:30:6f:24 via eth1
Aug 16 18:54:45 beer.pub.umich.edu dhcpd: DHCPREQUEST for 10.0.1.5 from 00:50:e4:30:6f:24 via eth1
Aug 16 18:54:45 beer.pub.umich.edu dhcpd: DHCPNAK on 10.0.1.5 to 00:50:e4:30:6f:24 via eth1
In basic terms, this is what was happening: a system comes online and tries to DISCOVER the DHCP server, the server OFFERs a good IP from it's pool, the client REQUESTed a bad IP, and the server does Not AcKnowledge that request. Then the cycle repeats. This made no sense to all three of us working on the problem. We tried restarting our DHCP server, we tried using a different server, etc etc. FINALLY, Sat realized that someone had an Apple AirPort they wanted to hook up. Searching the Apple Tech Info Library, we found out that the AirPort acts as a DHCP server! It was accepting all of the requests, and just giving out bad information! Took that off the network, and everything returned to normal. And it only took 8 hours to figure that out.
Aug 16 18:54:45 beer.pub.umich.edu dhcpd: DHCPDISCOVER from 00:50:e4:30:6f:24 via eth1
Aug 16 18:54:45 beer.pub.umich.edu dhcpd: DHCPOFFER on 141.211.148.76 to 00:50:e4:30:6f:24 via eth1
Aug 16 18:54:45 beer.pub.umich.edu dhcpd: DHCPREQUEST for 10.0.1.5 from 00:50:e4:30:6f:24 via eth1
Aug 16 18:54:45 beer.pub.umich.edu dhcpd: DHCPNAK on 10.0.1.5 to 00:50:e4:30:6f:24 via eth1
In basic terms, this is what was happening: a system comes online and tries to DISCOVER the DHCP server, the server OFFERs a good IP from it's pool, the client REQUESTed a bad IP, and the server does Not AcKnowledge that request. Then the cycle repeats. This made no sense to all three of us working on the problem. We tried restarting our DHCP server, we tried using a different server, etc etc. FINALLY, Sat realized that someone had an Apple AirPort they wanted to hook up. Searching the Apple Tech Info Library, we found out that the AirPort acts as a DHCP server! It was accepting all of the requests, and just giving out bad information! Took that off the network, and everything returned to normal. And it only took 8 hours to figure that out.