Oct 17 2021 06:35 PM
Oct 17 2021 06:35 PM
Hi Tech Community,
We are seeing some strange things on our deployment and am hoping someone here may have experienced the same and have found a workaround.
We have been using Microsoft Teams with a provider using direct routing for inbound and outbound calls to the PSTN. Things were working very well for the first month, however, it seems as though things have gone downhill since then. We are having issues where callers are placed in the queue that we have defined but notifications of this inbound call do not appear on the mobile or desktop applications. As such, our staff cannot pick the call up. This issue is with Microsoft support for now who originally blamed our SIP provider using an unsupported SBC version but now that this has been updated, Microsoft is finally escalating internally ...
In the meantime, we have noticed that when callers who don't show their caller ID call us, the inbound call notification only appears on the mobile app and not the desktop app. We have checked the desktop settings and even toggled this in case this helped but to no avail.
Is anyone else experiencing this issue?
Oct 18 2021 04:26 PMSolution
After almost 6 weeks of running what I would call a degraded phone system, Microsoft support have finally provided a recommendation that has resolved the problem. Whilst I do not have all the technical details of what is actually happening, it seems as though the "Conference Mode" makes a difference to whether the Desktop Teams App will receive notifications from callers that have blocked their Caller ID.
Oct 18 2021 06:18 PM - edited Oct 18 2021 06:23 PM
Just in case other people are experiencing the same issues as we were, I've decided to write up this post to go through our 6 week saga hunting down this issue in the hope that this helps other people as well as catch the attention of Microsoft who really need to sort out their customer service.
Whilst the overall attitude of the staff have mostly been very favourable, it was frustrating working with them when I felt powerless when it came to escalating the issue and even more annoyed when they resorted to finger-pointing at our SIP provider.
This deployment at this medium sized business with around 30 agents have a very simple set up where we have an Auto Attendant that presents the caller with an IVR giving callers a choice of joining one of three call queues. The call queues are configured using agent routing as the agents are doing mixed duties, serving the front desk as well as taking calls so agent routing works best for us. Agent routing is where all agents will be notified of incoming calls and have a chance to pick the call up. This may cause some race conditions but it is of minor impact to the staff whilst providing the best experience for the caller.
For whatever reason, Microsoft have not invested in providing decent Teams Calling reports. The report that was provided as an example is good starting point but the way the visualisations were configured were terrible as you would see things like 100% call abandonment that sticks out like a sore thumb even if it was just 1 call out of 1 in that particular time period. We pretty much had to create our own reports to provide actual useful information to our management staff.
We noticed that at the beginning of September, that the call abandonment numbers were significantly higher than the weeks before. We receive up to 80 calls per hour and the average number of abandoned calls went from 2 to 10. Whilst this doesn't seem significant, we conducted some call tests and found that there were times when we were in the queue but none of the agents were notified of the inbound call so they couldn't pick up. Some of our test calls went through immediately but some appeared to be clogged in the queue and then the queue would suddenly start flowing again and we would see a barrage of calls as if something was causing the queue to stall and then it would suddenly flow again.
The agents are all using the Teams Desktop App and Teams Certified headsets. After logging a ticket with Microsoft, we were asked to provide logs from Teams, enabling debugging then using the ctrl+alt+shift+1 key combo to generate the logs and after a week of going back and forth, as we were using direct routing through a SIP provider, we were asked to provide the SBC (session border controller) logs from our SIP provider.
After providing the SBC logs, we were advised that the SIP provider was not using a certified version as per https://docs.microsoft.com/en-us/microsoftteams/direct-routing-border-controllers. Now whilst this was a good reason for us to have our SIP provide update their version of the SBC, it does not take away from the fact that the problem was most likely on the Microsoft platform seeing that callers could actually get to the queue but it was the Teams desktop app that was not receiving a call notification. Despite pushing Microsoft support to have someone escalate and actually help us with the issue, they resorted to finger-pointing and would not assist until our SIP provide updated the SBC version. We pushed our SIP provider to look into updating their version and they said they needed to do the necessary testing before rolling it out.
Whilst this was happening, out of desperation, we had configured an additional queue and had callers move from one queue to the other after a time out period of 1 minute. That is, if a caller is in the first queue for one minute, they are moved into the second queue that was looked after by the same agents but new callers would land up in the first queue. Callers who were in the second queue for a minute would be pushed back into the first queue and callers would flip between queues till they were either answered or abandoned the call. This turned out to improve the situation somewhat. We had no idea why but it was not ideal and we were still seeing higher than normal call abandonment.
After waiting a little over a week for the SBC to be updated, of course, the issue persisted, as originally suspected, it was something to do with the Microsoft platform. We were asked to provide with a generic looking Microsoft website that instructed us to collect a bunch of logs and we yet again had to do multiple test calls until we were able to capture the situation where the issue would occur. After submitting the logs to Microsoft, yet another week goes by with nothing but "we are looking into it" responses when prompting them for an update.
Meanwhile, we started testing to see if the Teams Mobile app would provide any additional improvement as the Teams Desktop app appears to be far more problematic in many ways. That's when we found the smoking gun. As it turns out, most calls coming through the queue would appear on both the desktop and mobile apps but anonymous calls, where the caller had switched off their caller ID, would ONLY appear on the mobile app!
This explained everything. It explained why the queue would stall and then suddenly flow again and why having two queues with callers flipping between both queues every 1 minute actually helped the situation. The anonymous callers simply couldn't get through due to the lack of notifications to the desktop app and they would clog the queue until they hung up. The problem wouldn't have been as bad if we had used a different agent routing method but as I said before, it is the best for the business based on the current mode of operation.
So we decided to have a few agents install Teams apps on their phones so that they could use that to handle anonymous calls to prevent these calls from clogging the queue. I then updated the ticket with Microsoft with these findings and that was when it was suggested that we look into enabling "CONFERENCE MODE" in the call queue. After a full 6 WEEKS of running in a degraded mode, I'm glad to say that it appears to have solved the problem.
What really puts a bee in my bonnet is the poor customer service from Microsoft. It seems as though they needed everything from us. The logs, the test calls, everything. As the platform owner, they should have visibility of the entire call flow and be able to clearly see that it is the anonymous calls not being picked up. Instead of the support team taking ownership of the problem, it seems as though internal processes prevented them from escalating to other teams effectively and when our ticket as with engineering, they were more interested in seeing how they could palm the problem off to someone else. In the end, the problem was solved but through OUR OWN PERSEVERANCE and ongoing testing and trial and error to eventually isolate the exact cause and reporting it to Microsoft rather than support actually putting on the customer service hat and actually help with proper investigation of the issue.
Microsoft, as a US$2.3tn+ business and only getting bigger, you can do better.