Using photon::semaphore on AWS EC2, `signal` latency is high.

[lucaspeng12138]

Using photon::semaphore on AWS EC2, signal latency is high.

I have using photon::semaphore on AWS EC2. I found signal function may cost 300~600ms. I don't know what happens and why signal cost so long. I'm seeking assistance in understanding the cause and debugging strategies🙏.

[beef9999]

1. How many vCPUs are you using to synchronize with this semaphore? How many cores are there in your EC2.
2. Does this issue appear on other platforms or physical machines?

semaphore signal consists of lock and eventfd write (if across vCPU)

[lucaspeng12138]

@beef9999 my vcpu count is 12 and actually pod(deploy in k8s) limit 4 CPUs.
I get vcpu count by using photon::get_vcpu_num();

I didn't get same issue on other platforms such as Tencent Cloud.

[lucaspeng12138]

@beef9999 why my vcpu count is 12: I have three list of Executor, all of them length is 4. And they used to work as server, client and internal jobs.

[lucaspeng12138]

@beef9999 "How many vCPUs are you using to synchronize with this semaphore?" I think it is 1. I create a semaphore, and send into a std::thread, then wait it outside, such as following(project works in photon pool):

AsyncRetSharedPtr func A() {
    AsyncRetSharedPtr ret;
    _thread_pool->performance[ret]{
         do_something();
         ret->semaphore.signal();
   }
}

int main() {
    AsyncRetSharedPtr a = A();
    a->semaphore.wait();
}

[lihuiba]

@lucaspeng12138 Which branch and event engine are you using? How CPUs does the EC2 instance have?

[lucaspeng12138]

@lihuiba I'm using branch v0.7.1. Because we found it has better performance than later version.

EC2 CPU:

[lihuiba]

@lucaspeng12138 Could you try branch 0.8 and see whether the huge latency of 300~600ms exists or not? We had a major revision of semaphore since that branch, which may have fixed the issue.

BTW, could you talk more about the performance advantage of 0.7?

[beef9999]

I think the best practice is to set the number of vCPUs to be the same as the number of physical cores (of your EC2 instance). At least do not exceed it.

Since there is a spinlock in the Photon's lock implementation, I'm not sure if there will be performance issues when multiple OS threads are competing on a small set of physical cores.

[lihuiba]

@beef9999 He has 48 cpus in the EC2 instance.

[lihuiba]

@lucaspeng12138 Do you use epoll or io_uring?

[beef9999]

my vcpu count is 12 and actually pod(deploy in k8s) limit 4 CPUs

@lucaspeng12138 4 cpu in total or 4 * 12?

[lihuiba]

We had a major revision of semaphore since that branch

And we also had another revision of thread scheduler, which improved latency of cross-vcpu interrupt (wake up) of threads. So Trying 0.8 is highly recommended.

[lucaspeng12138]

@lucaspeng12138 Do you use epoll or io_uring?

@lihuiba I'm using epoll

my vcpu count is 12 and actually pod(deploy in k8s) limit 4 CPUs

@lucaspeng12138 4 cpu in total or 4 * 12?
@beef9999 4 cpu limits/pods, vcpu count is total 12/pods

[lucaspeng12138]

I think the best practice is to set the number of vCPUs to be the same as the number of physical cores (of your EC2 instance). At least do not exceed it.

Since there is a spinlock in the Photon's lock implementation, I'm not sure if there will be performance issues when multiple OS threads are competing on a small set of physical cores.

@beef9999 In our case, we have three pool, and it's difficult to use same one.
I may try this.

[beef9999]

@lucaspeng12138 Could you try branch 0.8 and see whether the huge latency of 300~600ms exists or not? We had a major revision of semaphore since that branch, which may have fixed the issue.

BTW, could you talk more about the performance advantage of 0.7?

@lihuiba Semaphore optimization is still in main.

@lucaspeng12138 You can try the main branch. And is there any way you can reduce your vCPU num?