New Nodes Become Unresponsive

[brockhaywood]

Wondering if you have any suggestions on the following.

I recently new nodes on our installation based on the ami build-nd-5. Upon initial deploy to them, they appear fine but after some period of time they no longer properly report their IP back to the shipbuilder server:

..
3:46 [sb-node7a] [s1_v473_egress_10002] - error retrieving ip
3:46 Retrying starting app dyno egress on host sb-node7a, failure reason: exit status 1
..

After the deploy, when inspecting the dynos on the node, everything looks fine. The dyno is running and I haven't found any problems with it with the exception that the shipbuilder server doesn't get the ip and thus doesn't update iptables/the loadbalancer to send traffic to it.

If you have any suggestions, it would be much appreciated.

[brockhaywood]

Follow up: it seems to be that the dyno is starting properly but it is taking a long time for it to report it's IP in the rootfs/app/ip file. I'm seeing this to be in the neighbourhood of 5 minutes.

[brockhaywood]

I've been able to narrow this down to networking taking a very long time to start.

The following commands take about 5 minutes to run:
root@s1_v474_egress_10001:# ifdown eth0
root@s1_v474_egress_10001:# ifup eth0

Are there any known configuration changes that should be made on the host node to improve the speed of DHCP?

[calebdoxsey]

Maybe this might help:

• First Ubuntu sets up networking like this: https://help.ubuntu.com/lts/serverguide/lxc.html#lxcbr0
• Next searching for issues with DHCP and LXC in ubuntu turned up some similar problems:
• This guy used a static IP instead of DHCP: http://ubuntuforums.org/showthread.php?t=2103163
• Could be a firewall problem (this is UFW, but perhaps iptables results in similar issues... there are some pretty crazy iptables rules in there): http://ubuntuforums.org/showthread.php?t=2106581

Personally I think I'd go the static IP route... You could do it similar to how ports were done.