We have a system setup on Amazon EC2's, divided between 2 availability zones. In our setup, we have a single RabbitMQ messaging server used by all servers.
Writes to a RabbitMQ queue that are cross-zone are 30% slower than queue writes that local within the zone hosting the RabbitMQ server.
From what I see in Amazon docs, the latency should be several milliseconds. But I am experiencing nearly 300 millisecond latency on cross-zone writes for a 60K block of data.
How can I diagnose the source of this delay? Our prime contractor controls the EC2 environment, so I am limited in what I can see/adjust/etc.
