SKYPE for BUSINESS and LYNC Troubleshooting Guide

SKYPE for BUSINESS and LYNC Troubleshooting Guide Version 1.0 Author: Thomas Poett, Microsoft MVP Lync © 2015 Skype for Business and Lync troubleshooting guide © 12.01.2015, Thomas Pött, Principal Consultant, Microsoft MVP Lync and PLSL 3rd level Support certified. Version 1.0 Contact: [email protected] Blog: http://lyncuc.blogspot.com The technical level of this document is 400. This article requires knowledge about Lync and Skype for Business in general. You need to know how to do configuration and all its related features. Lync and Skype for Business relay on several 3rd party components, as network or certificate authority, especially the CA is an important component for TLS encryption. This troubleshooting guide also focuses on external/ remote connection through the Edge server. Understanding of networking is crucial supping Lync/ Skype for Business. You need to be experienced with OCSLogger and SNOOPER. The document is structured in the general troubleshooting approach, digs deeply into SIP protocol and guides you through common issues. Note: Troubleshooting relays on your experiences from the past. You will become more advance how more often you do troubleshooting. Understanding of certain topic is still required. This guide will not go into the 3rd level support for Lync and Skype for Business component troubleshooting, e.g. MCU’s or Web Services. Contents Skype for Business and Lync troubleshooting guide ............................................................................... 2 Preamble and about the author .............................................................................................................. 6 Lync and Skype for Business Troubleshooting approach ........................................................................ 7 Environmental components ................................................................................................................ 8 Edge Server .......................................................................................................................................... 9 Conferencing Flow ............................................................................................................................. 10 Voice Call Processing ......................................................................................................................... 11 Support and troubleshooting tools ................................................................................................... 12 Client Tracing Log-File location: .................................................................................................... 12 Server Tracing Log-File location: ................................................................................................... 12 Converting Tracing Log-File location: ............................................................................................ 12 Service Site Logging (Central Logging Service): ............................................................................. 14 General information on TCP and SIP protocol ...................................................................................... 20 IP protocol ......................................................................................................................................... 20 TCP/IP protocol.............................................................................................................................. 20 UDP over IP protocol ..................................................................................................................... 22 TLS/ MTLS .......................................................................................................................................... 22 SIP protocol ....................................................................................................................................... 23 SIP protocol session setup ............................................................................................................. 23 SIP Commands: .............................................................................................................................. 24 SIP Message Fields:........................................................................................................................ 26 Simple SIP Call Setup ..................................................................................................................... 28 Session establishment and differences between IM, A/V and Conferencing ....................................... 30 Authentication internal and remote ................................................................................................. 30 Presence Query ................................................................................................................................. 30 IM Sessions ........................................................................................................................................ 31 Audio/Video Session (Desktop/ Application - Sharing) ..................................................................... 32 Conferencing ..................................................................................................................................... 33 When a call is escalated into a conference ................................................................................... 35 Lync Call Setup....................................................................................................................................... 36 Call Setup over EDGE Server (General) ............................................................................................. 36 Analyzing real world call setup .......................................................................................................... 37 INVITE the USER (OUTGOING) ....................................................................................................... 37 TRYING (INCOMIG) ........................................................................................................................ 40 SESSION PROGRESS (INCOMING) .................................................................................................. 40  PROGRESS REPORT (INCOMING) -2 times (identically send) ........................................................ 41 RINGING (INCOMING) – 4 times .................................................................................................... 41 PROGRESS REPORT (INCOMING) ................................................................................................... 42 PRACK (OUTGOING) ...................................................................................................................... 44 OK (INCOMING) ............................................................................................................................. 44 SESSION PROGRESS (INCOMING) .................................................................................................. 45 PRACK (OUTGOING) ...................................................................................................................... 46 OK (INCOMING) ............................................................................................................................. 47 OK (INCOMING) ............................................................................................................................. 48 ACK (OUTGOING) ........................................................................................................................... 50 INVITE (OUTGOING) ...................................................................................................................... 51 TRYING (INCOMING)...................................................................................................................... 52 OK (INCOMING) ............................................................................................................................. 53 ACK (OUTGOING) ........................................................................................................................... 54 UPDATE (OUTGOING) .................................................................................................................... 55 OK (INCOMING) ............................................................................................................................. 56 BYE (INCOMING) ............................................................................................................................ 57 OK (OUTGOING)............................................................................................................................. 58 Troubleshooting IM, Calls with A/V....................................................................................................... 59 AV Address Exchange, negotiation of candidates ............................................................................. 59 Audio Video Call failed with ms-client-diagnostics (one client is external): ................................. 62 Audio Video Call failed with ms-client-diagnostics: (both client are external): ............................ 65 Diagnostic headers ............................................................................................................................ 67 MS-DIAGNOSTICS .......................................................................................................................... 67 MS-CLIENT-DIAGNOSTICS .............................................................................................................. 70 Monitoring Reports and Call Quality Issues ...................................................................................... 73 Example: Submitting Metrics after Conference call ...................................................................... 76 Software Defined Networking (SDN)............................................................................................. 79 Preventing Configuration and other Issues (Testing Commands) ..................................................... 80 IM................................................................................................................................................... 80 Voice .............................................................................................................................................. 81 Conferencing ................................................................................................................................. 83 WEB Services ................................................................................................................................. 84 EDGE (external/ remote) ............................................................................................................... 85 Health Monitoring Test User ......................................................................................................... 85 Troubleshooting Exchange Integration ................................................................................................. 86  Verify Exchange AutoDiscover setup................................................................................................. 87 Exchange Unified Contact Store Integration ..................................................................................... 88 Exchange IM integration on Outlook Web Apps ............................................................................... 91 Exchange Web Service Integration.................................................................................................... 92 Exchange Unified Messaging Integration .......................................................................................... 93 Two more important troubleshooting task have to be validate. .................................................. 94 Troubleshooting conferences................................................................................................................ 96 Persistent Shared Object Model (PSOM) protocol............................................................................ 97 External FQDN with single IP address: .............................................................................................. 98 External FQDN with multiple IP addresses: ....................................................................................... 99 Conference INVITE and ACCESS....................................................................................................... 100 Call flow explanation to the illustration above ........................................................................... 101 Why not Single IP on EDGE Port 444 Problem….............................................................................. 102 Client doesn’t open Lync when meeting link is clicked. .................................................................. 108 Validating Conference Settings and Expiration ............................................................................... 109 Activation and Deactivation ............................................................................................................ 110 Resetting a default Conferencing ID................................................................................................ 112 Troubleshooting Lync and Skype for Business Web Services .............................................................. 115 Internal and External Web Services IIS............................................................................................ 115 Mobility Services (for mobile clients) .............................................................................................. 118 Scenario 1 (internal mobile/internal full client): ............................................................................. 119 Scenario 2 (internal mobile behind internal firewall/internal full client): ...................................... 120 Scenario 3 (internal mobile/external full client): ............................................................................ 120 Having a look into the discovery and logon process: ...................................................................... 121 Lync 2010 Mobile App: ................................................................................................................ 122 Lync 2013/ Skype for Business Mobile App (Windows, iPhone, iPad and Android) ................... 122 Address Book Web Services for Mobile Devices ......................................................................... 123 Troubleshooting Office Web App Server............................................................................................. 125 Enterprise Voice .................................................................................................................................. 128 Voice Route and Trunk parameter .................................................................................................. 128 References ........................................................................................................................................... 130 Preamble and about the author First I have to say thanks to my wonderful wife supporting me during the writing and to my actual company I’m with. I had to spend some time writing on this free eBook, which consumed quite an amount of my time with my family. This eBook is about troubleshooting Skype for Business and Lync. A complex solution in unified communication making people’s life more simpler, connecting to other at any point of time, staying in contact with fellow friends and family members. Planning and build UC solution is only on site of the coin. Understanding how this technology works is the other side. Developing a set of skill supporting and analyzing issues in this environment is even more advanced. Therefore I decided, after I receive many inquiries, supporting my fellow blog reads in troubleshooting. Writing a guide not only focusing on troubleshooting procedures, instead I explain the complexity in this area. It is essential for troubleshooting to understand where, or at which point within a communication path the issue might have occurred. Thomas Poett (Author and Microsoft MVP for Lync) Professional, consistent, and experienced expert who is technically savvy with over 20 years of experience in IT, telecommunication and software development. Additional extensive experience in business and market development. Specialized in intercultural and business relationship in Asia. Successful in providing leadership on new topics and complex global projects that require interfacing with internal/external teams and ecosystems. Early adaptor of visionary technologies. He is awarded as a Microsoft MVP for more than 3 years, sharing Lync knowledge and guidance for planning processes. I achieved the Premier Support for Lync Partners (PSLP) certification and support the teams for Lync 2010/ 2013 3rd level troubleshooting. Special thanks to: Jeff Schertz (Polycom), Richard Brynteson (MVP), Thomas Binder (Microsoft) and Johann Deutinger (Ferrari electronics AG), my Allgeier workmates for their support and information provided personally or via their blogs. Lync and Skype for Business Troubleshooting approach Seeing troubleshooting from all perspectives, we need a matrix where we are enabled analyzing the area which possibly can causes any issues. As identified, we see 4 major and a common configuration area. The areas are (Quality issues):  Network  Core Performance  Gateway  Devices The area of configuration (environment setup):  Voice Setup (from Dial Plans until Normalization and Routes)  Gateway configuration  Exchange Unified Messaging integration Making your troubleshooting approach faster. Here is a short approach of the most common issues. NOTE: If you are facing an issue with AV not working externally check the following 1- PORTS (This is normally the issue) 2- DNS Records 3- Certificates and trusts Environmental components Since I spoke about the network, let’s see what else can be identified: NETWORK SERVER CLIENT WAN MIS-Configuration Application Settings Router / Switches (Enterprise Voice/ DNS) (Client CU s) Bandwidth Server Resources Configuration Firewalls Connectivity Type of network (wired/ wireless) Seeing here the three essential areas of involved components. Mostly, after you had reviewed the involved server, which goes along with the configuration, you see the issues related to your network. This is why we highly emphasize the importance of a network assessment and the implementation of SDN, respective the implementation of network monitoring. Coming once back to the configuration. Not only is the Enterprise Voice afflicted with configuration issues, so to DNS and Exchange integrations. That’s why it is important, you have a proper environment planning done upfront. Configuration issues can be therefore identified during a conceptual review. Network is what matters most. Therefore you need to understand the reliability of networks. Your LAN is more reliable than your WAN, while the Internet is the most unreliable network. During troubleshooting, you have to identify this location where the issue occurred. Edge Server Two important aspects are understanding the flow and processing of conferencing, voice and once more the Edge server: INTERNET DMZ CORPORATE HTTPS (443) HTTP (80) HTTPS(4443) Ext. Proxy Service External IP HTTP (8080) External Web Services Reverse Proxy Int. Proxy Service Internal IP Internal Web XMPP (TCP:5269) XMPP Proxy Service Services HTTP (Public CA CRL Check) CLS/MTLS(TCP:50001) SIP/TLS (TCP:443) CLS/MTLS(TCP:50002) Lync Services SIP/MTLS(TCP:5061) CLS/MTLS(TCP:50003) Access Edge Service Edge DNS(UDP/TCP:53) External IP XMPP/MTLS(TCP:23456) Internal IP SIP/MTLS(TCP:5061) PSOM/MTLS(TCP:8057) OAuth Service PSOM/TLS(TCP:443) Lync SIP/MTLS(TCP:5062) WebCon Edge Service Edge A/V STUN(UDP:3478) External IP Authentication STUN(TCP:443) STUN(UDP:3478) Service HTTPS(TCP:4443) STUN(TCP:443) Lync Edge RTP(UDP/TCP:50.000-59.999) AV Edge service External IP In case if I repeat this statement, please do not use Edge server with a single IP address. Segregate the three different service by an individual IP address. Please understand the TCP/ UDP port openings and the related packet direction. This is what matters during your firewall setup. Conferencing Flow Microsoft implemented their own conferencing protocol, call CCCP or C3P (Conference Control Channel Protocol/ Centralized Conference Control Protocol), based on Framework for Conferencing with the Session Initiation Protocol. [RFC4353] Reference: http://msdn.microsoft.com/en-us/library/cc431498(v=office.12).aspx Within a conference, 4 different MCU are existing. During troubleshooting you might need to trace some or all MCU’s with OCSLogger/ CLS. This provides you the internal view what is going on inside. Tracing the protocol, since the clients are joining a conference via SIP, you trace as usual. In the prospective of a client joining a conference, you will see the C3P over SIP only. Voice Call Processing With Enterprise Voice topics it is difficult to start with. Nevertheless, the first always is the understanding of the call process and it’s related flow with all involved components. While a voice call initiated with a SIP URI is immediately processed, the call using a dialed number follows an entire different flow. As we can see in the call processing flow, the second decision is made where the call is identified as an E.164 call, a call starting with a “+” sign. If not the number will be normalized. Again here, please make sure it is always E.164. Once the normalization is done, the important Reverse Number Lookup takes place, where the number is search for a matching user, either AD or Outlook. This enables the client to display the user name instead of a phone number. If the user is identified as an internal user enabled for UC, the call will be processed via it’s SIP URI. Only form here the call processing in direction to external will start. It involves the checkup for invalid numbers and Call Park Orbits, processed to now by the Voice Policies with their PSTN Usage Records and finally the Routes with all their configurations. After this processing the call is passed over to the Mediation Server and the related TRUNK configuration. Initiated LyncCall SIP URI User=phone Dial Plan Normalization Rule NO NO Normalization Rule E-9-1-1? Global? Normalization Rule YES YES 404: No Call Park Orbit Range matching rule Dialing Reverse Number Lookup behavior Routing & MATCH NO MATCH Authorization Location Policy Routes 3.Voice Policy PSTN Usage Route 1. Vacant Number Range PSTN Usage Route Route PSTN Usage PSTN Usage 2. Call Park Orbit Route Mediation Server and Trunk Configuration 403: No Route Announcement or found Call Park Application Inbound Routing Gateway / IP-PBX / SIP Trunk Lync Endpoint Receives Call External Endpoint Receives Call If this process/ flow is correctly configured and ran along, the next troubleshooting step should occur on the Trunks, Gateway and other involved SBC components. I have dedicated an entire chapter regarding Enterprise Voice troubleshooting. Support and troubleshooting tools Which tools can be used for analyzing? Most important is SNOOPER, you need to trace the SIP messages. NETMON is even important if you want to look in front of SIP, into TCP. Here you most best analyze the impact of firewalls or routers. Other tools are:  ICE Warning Flag Decoder  PreCall Diagnostic  (Central Logging Service)  (Audio Test Service)  In CSCP: Voice Routing Test Case Client Tracing Log-File location: The Lync 2010 client tracing logs are located at: %userprofile%\tracing. The Lync 2013 client tracing logs located at: %userprofile%\appdata\Local\Microsoft\Office\15.0\Lync\Tracing Server Tracing Log-File location: Both, for Lync Server 2010, Lync 2013 and Skype for Business the log-file location is c:\windows\tracing Once he log file is converted and visible in Snooper, the temporarily generate log fine (txt format) will be in the user profile %userprofile%\appdata\Local\Temp\OCSLogger_xxxx_xx_xx_xx… Converting Tracing Log-File location: The CLS and OCSLogger use event log format (*.etl) those files cannot be opened. They are compressed in ratio 1:5. You need to convert those etl files to txt. You only can do this manually from the command line. The exact file location must be provided after the “package for PSS” (/pss) option. C:\Program Files\Microsoft Lync Server 2013\Debugging Tools>OCSLogger.exe "/pss:c:\windows\tracing" This will open OCSLogger, where you need to choose the “etl” files by pressing the “Analyze Log Files” button. In the next windows, the possible (found) log files will be displayed. Where after you can view the log in Snooper for further analyses. NOTE: The OCSLogger depends on the server version. You need to use OCSLogger in the command line from the correct server source. Else the file will not be converted properly. A mismatch can be identified, if the result in Snooper look like the following illustration. Service Site Logging (Central Logging Service): In lager deployments or even if you run the servers in a pool setup, the logging can become a hassle in case of finding the correct server, where the troubleshooting needs to be started. If you have multiple servers in a pool, you can’t identify the individual server, where a e.g. failing call is processed. We need to make sure we can trace all activities across those member servers. Another example is, if you have multiple pool, like several front end pool and mediation pools, you might also want to trace the path a session is running along. If we now start using the OCSLogger on all those machines, we have problems consolidating all so log and as well we do have difficulties starting multiple OCSLogger session nearly simultaneously. The solution here is: Centralized Logging Service. A service for controlled collection of data, with a broad or narrow scope. The service is setup with two components, the CLS Service Agent (listening on incoming command on TCP port 50001, 50002, 50003) and the CLS Service Controller. NOTE: You should entirely learn about CLS. I will provide a generic overview helping you to make use of CLS. Elements of Central Logging Service Similar to OCSLogger, we will find those setting for CLS. In this chapter I have used the Microsoft Technet Examples making the understandings of CLS more transparent. There are three kinds of CLS elements:  Providers are the COMPONENTS in OCSLogger  Logging levels OCSLogger provided the option to choose a number of levels of detail for the data collected.  All of type fatal, error, warning, and info  Fatal messages that indicate a failure  Error messages that indicate an error, plus fatal messages.  Warning messages that indicate a warning for the defined provider, plus fatal and error messages.  Info messages that indicate an informational message for the defined provider, plus fatal, error, and warning messages.  Verbose messages of type fatal, error, warning and info for the defined provider.  Flags defined what type of information could retrieve  TF_Connection information about connections established to and from a particular component  TF_Security events/log entries related to security. For example, for SipStack, these are security events such as domain validation failure, and client authentication/authorization failures.  TF_Diag diagnostics events like DNS warnings/errors.  TF_Protocol protocol messages like SIP and Combined Community Codec Pack messages.  TF_Component components specified  All Sets all available flags available for the provider. Scenario for Central Logging Service A scenario include the aforementioned elements and define the scope of logging. The scope can be either a computer, a pool, sites or global. However you can only maximum two different scenarios for any given scope at any given time. In Lync or Skype for Business management shell, you must provide an identity addressing configurations. This identity defines the scope in CLS. e.g. –Identity “site:Europe/LyssServiceScenario” –Provider $LyssProvider or –Name “LyssServiceScenario” – Parent “site:Europe” –Provider $LyssProvider As we have seen, the Provide is defined as a string, this is because of the provider has to be configured the following way too: $LyssProvider = New-CsClsProvider -Name "Lyss" -Type "WPP" -Level "Debug" - Flags "TF_Connection, TF_Diag" The process of working with an CLS Scenario follows the principle of: New-CsClsScenario -Name "SIPStack" -Parent "site:Europe" -Provider $SIPStackProvider After creating a scenario, can further modify is: Set-CsClsScenario -Identity <name of scope and scenario> -Provider @{Replace=<providers to replace existing provider set>} If you need to remove a scenario, this will be done by: Remove-CsClsScenario -Identity <name of scope and scenario> Removing or adding a provider to existing scenario uses the Edit-CsClsScenario: Remove: Edit-CsClsScenario -ScenarioName <scenario to edit> -ProviderName <provider to remove> -Remove Add: Edit-CsClsScenario -ScenarioName <scenario to edit> -ProviderName < provider to add> -Level <type level> -Flags <type flags> Having a look into the preconfigured scenarios, which are likely to be sufficient for the beginning: Get-CsClsScenario | fl *ident* Additionally, we should have a look into the provider, which provides the information about the component its level and flags. Get-CsClsScenario | Where-Object {$_.identity -like "Global/AlwaysOn"} | Select-Object provider | Select-Object - ExpandProperty provider Configuration Settings for Central Logging Service Before we can start, stop, flush or search results from CLS, we need to have the configuration for CLS defined. As in the same way of scenarios, we can choose to define different setting over different scopes (Global or Site). The illustration below, should help you define those settings: Command in management shell are:  Set-CsClsConfiguration  Remove-CsClsConfiguration  New-CsClsConfiguration  Get-CsClsConfiguration Start and Stopping Scenarios for Central Logging Service Starting and Stopping must be configured from the management shell. It is recommended in troubleshooting using advance paramters. Start-CsClsLogging -Scenario UserReplicator -Duration 8:00 -Pools "pool01.contoso.net" The default scenario is ALWAYSON, logging all relevant level of information and cycles the log files. If you have this scenario started, at any given point of time you are enabled having a look into the log files and extract what is need (see next chapter about searching) Start-CsClsLogging -Scenario AlwaysOn Stopping the CLS is available with Stop-CsClsLogging -Scenario AlwaysOn Analyzing which scenario is running simple type: Show-CsClsLogging Searching in Central Logging Service Searching the log file is most crucial for troubleshooting. Whenever you need information make sure you know what your are looking for. The Search-CsClsLogging is the powerful command helping you extracting not only information from a single computer. It is more extracting information about an entire path or even the entire environment. Not enough, if can also filter based on IP-Addresses or URI, components Sip Contents and more. Example: Search-CsClsLogging -pool "sykpe-pool.contoso.com" -IP "192.168.0.242" -Uri "sip:[email protected]" -MatchAny Here is a table of all parameters you can include in your search. Parameter Description CallId Call identifier for specific call. Components list of components. Computers list of the computers ConferenceId Conference ID CorrelationIds list of correlation IDs to search EndTime Specify local time zone. Defaults to 5 minutes after current time if no StartTime specified, otherwise defaults to 30 minutes after StartTime -StartTime "8/31/2012 8:00AM" IP IP address LogLevel minimum type of log entry MatchAll all the included criteria must be matched. MatchAny only one of the included criteria must be matched. This is the default setting, similar to a OR command OutputFilePath Defines the log file search result as text file to the specified location and name. Otherwise they are written to the console. Phone Phone number to be searched for. It must match E.164 format!. Pools Comma-separated list of the pools SipContents Arbitrary text to search for within the body of a SIP message. SkipNetworkLogs instructs the Search-CsClsLogging cmdlet to avoid searching network logs. StartTime Beginning date and time for the log entries to be searched. Specified in local time zone. Defaults to 30 minutes before EndTime. Uri Uri to be searched for. Note: The best possibility for an end-to-end trace of SIP session is provided if you use the Centralized Logging Service. You are entitled drawing an end-to-end session flow chat. This helps you verifying a SIP session and other relevant data helpful troubleshooting Lync and Skype for Business. General information on TCP and SIP protocol Before you start troubleshooting or build your skill for troubleshooting, the basic understanding how the underlying protocols are working is essential. First we start with the IP protocol, while a TLS/ MTLS inside view will be discussed. Finally the SIP protocol is the most essential for troubleshooting. NOTE: In troubleshooting the entire knowledge about the 7 layer ISO model is required. You need to identify where possible issues are to locate. ISDN has the same layer approach, therefore identifying e.g. if it is a connection or configuration issue, you need the understandings of all this dependencies. IP protocol In Lync/ Skype for Business, we make use of two ISO layer of IP, TCP (Transmission Control Protocol) and UDP (User Datagram Protocol). TCP/IP protocol Generic in Lync and Skype for Business, all communication runs of TCP. This includes the internal and external IP traffic. TCP is also always a fallback path for Audio and Video data if the UDP path has issues or fails. Non expected event CONNECT/SYN (Step 1 of 3-way-handshake) Session CLOSE Start CLOSE/- Path client/ receiver Path server/ sender LISTEN/- CLOSE/- SYN/SYN+ACK (Step2 of 3-way-handshake LISTEN RST/- SEND/SYN SYN SYN RECEIVE SENT SYN/SYN+ACK (simultaneously open) State of transmission ACK/- SYN+ACK/ACK ESTABLISHED (Step 3 of 3-way-handshake) CLOSE/FIN FIN/ACK FIN/ACK FIN WAIT 1 CLOSING CLOSE WAIT FIN+ACK/ACK ACK/- ACK/- CLOSE/FIN FIN WAIT 2 FIN/ACK TIMED WAIT LAST ACK Active session closing Passive session closing TIMEOUT ACK/- CLOSED The drawing gives the entire overview of the process how TCP transmission will occur. If you are using Microsoft NETMON, you can entirely trace the TCP transmission between the sender and receiver. During troubleshooting an identified often issues on firewalls, where entire or only a single direction was blocked. You need the approach to identify, which path drops packets or also which packet got lost. The path a packet is running is relevant too, especially if you have firewalls in place. Trace always that the sending and receiving stream is running the EXACT same path. Sometime in routed environments you will find that source and destination network is routed differently from the prospective of sender and receiver. I give you an example in the next illustration, where the path is routed differently as you can see. This is a very common mistake. Not only inside the LAN, especially in the DMZ/INTERNET setup. ROUTING: 10.10.20.0/24 GW 10.10.10.254 10.10.10.254 WRONG PATH TCP ACK 10.10.20.1 CORRECT PATH TCP ACK 10.10.10.1 TCP SYN 10.10.20.254 ROUTING: 10.10.10.0/24 GW10.10.20.254 The Sender Receiver Setup follows the first initial TCP setup as illustrated in the next table: SYN-SENT → <SEQ=100><CTL=SYN> → SYN-RECEIVED SYN/ACK-RECEIVED ← <SEQ=300><ACK=101><CTL=SYN,ACK> ← SYN/ACK-SENT ACK-SENT → <SEQ=101><ACK=301><CTL=ACK> → ESTABLISHED You can also see the SEQ (Sequence Number). This is where the packet order comes into the place. UDP over IP protocol The UPD transmission is quite identically with TCP, beside it is not RELIABLE. Meaning, the sender do not “care” if the client received the send packet. It just keeps streaming. This explains why Audio and Video data is best to be UDP. As we learned from our CD/ DVD players, this data can be “error corrected”. It doesn’t really matter if we have some streaming information lost. Same applies to Lync/ Skype for Business. We can transmit AV data over unreliable networks, e.g. internet or Wi-Fi. When the packet lost reach certain level, first we drop the call quality, if the lost rate is still increasing, the connection might be terminated. Another subject of matter in UDP is, the order in which the packets are flowing in at the receiver side do not matter, since there is no control in place and process bringing them into order. Lync and Skype for Business with all their codecs do never start a UDP communication if the sender and receiver didn’t agree of using UDP. The AV session establishment will always be TCP first. Why? We need to negotiate a lot of upfront setting, e.g. the chosen protocol/ path, the codec and more. Only after the negotiation, the UDP dataflow starts. TLS/ MTLS Most common mistake during encryption. You can’t verify this often enough. Lync and Skype for Business is “SECURED by DESIGN!”, no communication ever goes unencrypted. Both server and client must just certificated based encryption. Authentication also relay on certificates, after initial NTLM/ KERBEROS authentication took place at the very first connect. The TLS-DSK technology, where a Lync/ Skype for Business server act’s as a certificate authority, handling the clients personal, per user base certificate over and the client stores this certificate in its local store. This is also the only certificate NOT having any trusted root authorities required. Since the Lync/ Skype for Business authentication service can himself identity the certificate. Make sure during troubleshooting, that this certs are present on a client site and valid (date). The period can be set per Server. All other communication internally, as well externally relays on privat and public certificate authorities (CA). Where the certificate need the correct CN/ SN, the root CA must be in the Trusted Root Authority store. Please refer here to my blog about certificates used with Lync. Explaining the difference between TLS and MTLS can be consolidated into: While a TLS connection is session oriented, the MTLS secured connection can handle multiple session in parallel. SIP protocol With SIP protocol we came finally into the first real troubleshooting aspects for Lync and Skype for Business. Once we had verified that TCP/IP and UDP is working correctly, we must have a look into the communication itself. Since we understand, the entire traffic is encrypted, we cannot use NETMON anymore. We would only see TLS communication flying around and don’t really see anything related to the SIP communication. Well, we can identify the destination ports and can assume now it SIP or it CCCP. Here the CLS (Centralized Logging Service), OCSLogger and Snooper is our tool of choice. Only about the snooper and all it parameter we could make an entire technical reference. Since we want to focus on troubleshooting and the main issues, as well as the approach of troubleshooting. The core components are SIPStack and S4. Just if we analyze SIP, the SIPStack is our favorite. (in 3rd level support you need Snooper and CLS for very detailed analyzes even down to the Server components.) If you utilize Enterprise Voice, please be aware under all circumstance make use of E.164 number format. Learn everything about how to plan, setup and implement Enterprise Voice. Regarding this topic please ready my Demystify Enterprise Voice article. In Enterprise Voice you not only troubleshoot SIP, more like you have to troubleshoot the number format, e.g. why Exchange UM is not doing reverse number look up or why a GW destination isn’t reached. SIP protocol session setup Before we are having a look into a SIP communication, you need to understand SIP communication. Similar with a TCP session establishment and handling, SIP follows this approach too. Additionally to the login and register workflow, the SIP communication basics should help you understanding voice based solutions on Lync and Skype for Business. If you initiate an IM Session the flow is with a call setup. The provided workflow can be seen as identically if the entire call setup runs over multiple systems involved, e.g. Client, Server, and Gateway, hopping through all parties. Part of those workflows are the understandings of SDP, ICE and Early-Media. Here the path can be tracked in the VIA or the ROUTE header. The illustration below shows a successful established call between the CALLER and CALLEE. The Caller initiates the call be sending an INVITE to the Callee, who then returns the information of RINGING and OK. After the receiver of those both commands, the Caller must acknowledge this action by sending an ACK. This than after the Callee has hook off the call the RTP media starts to be transmitted, as voice session is established. The site now hanging up will send the BYE command, which has to be acknowledged by 200 OK. SIP Commands: SIP commands describe the session setup information. They are the core in SIP communication. While in the next chapter, the message fields are send along with the SIP command, provide more detailed and necessary information. If you use SNOOPER, SNOPER has the capability displaying the CALL FLOW based on the SIP Commands. INVITE (https://datatracker.ietf.org/doc/rfc4235/?include_text=1) Command that is starting all dialogs, Calls, Presence and IM. Dialogs can be theoretically created by many different methods, although RFC 3261 defines only one: the INVITE method. RINGING (https://datatracker.ietf.org/doc/rfc3960/?include_text=1) An acknowledgement send informing about the target is ringing. Also related to Early-Media. SESSION PROGRESS (http://www.ietf.org/proceedings/46/I-D/draft-ietf-sip-183-00.txt) Addressed to the RINGING and the related SDP Message. Enhancing the RINGING with further information. OK Simple protocol related acknowledgement on any command needed to be accomplished. PRACK (https://www.ietf.org/rfc/rfc3262.txt) A similar command like the BYE, but not acknowledging. A provisional response on the INVITE. It will be marked by a RSeq, referring to the related command send earlier (e.g. INVITE). A sub command within the PRACK is RAck, it response to support reliability of provisional responses ACK Command acknowledging the progress made. Related to the SIP protocol RFC. BYE Termination command for ending a SIP session. SIP Message Fields: The SIP messages fields are your gateway for identifying what will and what is going on and represent how the communication flow will be established. This short description is supported by the reference to the originated RFC. It provides you the most simple and fastest understanding of SIP protocol troubleshooting CALLER-ID: Unique identifier for each call (best for grouping calls in Snooper) AUTHENTICATION-INFO: This field provides information about the possible and choose authentication method, e.g. NTLS.KERBEROS or TLS-DSK. VIA: The path the SIP message run along, providing the path from the source to target Record-Route This field is similar to the VIA field, but contains information about the FQDN. FROM: “display name”<SIP Address> and tags + identifier A SIP address either start with SIP: for a sip call or TEL: for phone call TO: Target e.g. user, phone or application P-ASSERTED-IDENTITY: https://www.ietf.org/rfc/rfc3325.txt The PAI header provides a way to verify the identity of the caller. Regarding those settings, you need to understand the SIP Trunk configuration in Lync/ Skype for Business: http://technet.microsoft.com/en-us/library/jj688104.aspx ALLOW: This lists the “allowed” SIP commands usable with in this session. CSeq/ RSeq/ RAck: An increasing number starting with the first command, mostly INVITE, the CSeq rever also back in other command, which work as a response to the initial command. RSeq and RAck, are similar to CSeq, but act with in sequence as a “sub-counter”. User-Agent: Identifies the client type, e.g. Lync client, a phone edition or even the Server Application itself. UCCAPI/15.0.4667.1000 OC/15.0.4667.1000 (Microsoft Lync) ms-diagnostics/ ms-client-diagnostic: The most important message, client and IP/ Port information are provided with in the statement. You can analyze why this action in sequence was chosen, e.g Call terminated by a user, or other network related causes Ms-user-logon-data: e.g. RemoteUser, identifies, from where the user is logged in ms-media-location-type Within the SIP message is identified, which network the client is in. Supporting the choice for matching candiates. a=candidate Every client can be position in different networks, either in LAN (corporate or private home), could be in the Internet. The candidate show the possible connection method, if TCP or UDP and which IP addresses are involved, LAN or behind NAT. This is the information clients need knowing the best possible path connection media data. SERVER: An information field, if a Skype for Business Server or client is sending this SIP message. Possible are also Applications. Most interesting is the client version, either Lync client, or mobile device. This helps us identifying the talking/ sending device of the related message. There are many more options included in SIP messages, but fact is for 90% of all troubleshooting cases the documented fields and commands are sufficient enough. Simple SIP Call Setup In one of the chapters aforementioned, the session establishment and closing was described for the TCP/IP protocol. If we compare those establishments, we will find similarities valid for SIP session establishment too. Within the next two paragraphs, we are digging into a Call setup, explain at which point in a session, media data streams are establishment and what early-media is. Without Early-Media This “normal call setup” relates to the basic session establishment, Caller Callee | | | INVITE | |------------------------------>| | | | 100 Trying. | |<------------------------------| | 101 Session Progress | (Report) |<------------------------------| | 180 Ringing | |<------------------------------| | 200 OK | |<------------------------------| | ACK | |------------------------------>| | Both Way RTP Media | most likely UDP Data if possible |<=============================>| | BYE | |<------------------------------| | 200 OK | |------------------------------>| | | Lync and Skype for Business have different scenarios, where the path of AV data is different. So please keep in mind, if two clients are in a same or directly routed network segment (without NAT), the media data stream is established always between the both clients, regardless if you are within the same Topology or Federated Partner. IM session indeed have a server involved, IM cannot be transferred directly between clients. Same is valid if we have a multi-party call, where the Server MCUs handle the AV distribution. NOTE: Making a troubleshooting approach successful, first you should stress what is expected. Meaning here from where to where the call flows, which components (Servers) are involved. If a user is busy and can’t accept a call. A message of 486 Busy Here instead of 180 Ringing is presented to Caller. The Caller send a BYE to the Callee and the session is aborted. With Early-Media in Lync Modern SIP environments support a faster call setup. This requires the both client starting data exchange earlier, before a final IP path was negotiated. This is called early-media. Where the audio/ video session is established before the called party takes the call. Early Media and Ringtone generation is described in the RFC 3960. http://www.ietf.org/rfc/rfc3960.txt Allowing early media in a SIP call, we must have an INVITE and 183 Session Progress command being send and this command contains the SDP data (Session Description Protocol). Caller Callee | | | INVITE | (contains SDP information - caller) |------------------------------>| | | | 100 Trying. | |<------------------------------| | 180 Ringing | |<------------------------------| | 183 Session Progress | (contains SDP information - callee) |<------------------------------| | 101 Progress Report | |<------------------------------| | PRACK | |------------------------------>| | Both Way RTP Media | EARLY MEDIA starts flowing A/V |<=============================>| | 200 OK | |<------------------------------| | ACK | |------------------------------>| | Both Way RTP Media | most likely UDP Data if possible |<=============================>| | BYE | |<------------------------------| | 200 OK | |------------------------------>| | | Different compared with the first session initiation is the early-media involvement. Instead of a 101 Session Progress replay, we need to include more information from the called target. That’s why it sends an enhanced 183 Session Progress, which contains the Session Description Protocol (SDP) in the 101 Progress Report message. This SDP is used to establish a media connection that carries those network tones and messages. Immediately after the call was taken (you hock off the phone/ call) the media data can be transferred without any delay. The acknowledgement will not be a ACK, instead it replies with PRACK and the media (audio) can start flowing even if the session is not fully established. The rest of the session follow the identical flow like the first illustration above. Starting with the 200 OK. Session establishment and differences between IM, A/V and Conferencing For troubleshooting it is recommended that you fully understand the different types of session establishments. The behavior for example between an IM Session and an A/V call is quite different. In case you need to support issues, it is essential to know where to identify and where to start with your support approach. In general we differentiate between server involved session, either in one-way or two-way, as well MCU (Multipoint Connection Unit) or peer-to-peer connections Therefore we have a look into the different types of communications. Authentication internal and remote 3. Server presents the certificate to Edge Server Authentication 4. Edge presents 7. Trusted and encrypted certificate to Client connection established 1. After DNS resolution, Client contacts the Lync Edge Access Server. 2. Edge Access Server connect to Director 5. Client authenticates Server (Next Hop) 6. Authentication is processed Presence Query Presence is a one-way query, meaning here that the user who want to add presence to a contact either in his contact list, or when he was opening a communication window, send a query, the SUBSCRIBE out to the referred target. This message contains an EVENT called presence (“yellow”) and SUPPORTED of “ms-benotify”. As well as a XML batch is sent containing the query inbetween the “action name” One Way problem of Presence and IM… One user can the other not…. SUBSCRIBE sip:[email protected] SIP/2.0 Via: SIP/2.0/TLS 192.168.0.16:61813 Max-Forwards: 70 From: <sip:[email protected]>;tag=a2ed804245;epid=0639570a7f To: <sip:[email protected]> Call-ID: f7bb816122e24b68b352d07413f063e8 CSeq: 1 SUBSCRIBE Contact: <sip:[email protected];opaque=user:epid:2b02BQuhtlW-2_O2vbNPYAAA;gruu> User-Agent: UCCAPI/15.0.4675.1000 OC/15.0.4675.1000 (Microsoft Lync) Event: presence Accept: application/msrtc-event-categories+xml, application/xpidf+xml, text/xml+msrtc.pidf, application/pidf+xml, application/rlmi+xml, multipart/related Supported: com.microsoft.autoextend Supported: ms-benotify Supported: ms-piggyback-first-notify Proxy-Require: ms-benotify Proxy-Authorization: TLS-DSK qop="auth", realm="SIP Communications Service", opaque="6B133F97", targetname="xsrvlync7.domain-a.local", crand="ae97593f", cnum="17", response="43890d90c8fb20d9d4776370dd874f34f71c845c" Content-Type: application/msrtc-adrl-categorylist+xml Content-Length: 478 - <batchSub xmlns="http://schemas.microsoft.com/2006/01/sip/batch-subscribe" uri="sip:[email protected]" name=""> - <action name="subscribe" id="1784768368"> - <adhocList> <resource uri="sip:[email protected]"/> - </adhocList> - <categoryList xmlns="http://schemas.microsoft.com/2006/09/sip/categorylist"> <category name="state"/> <category name="services"/> <category name="note"/> <category name="contactCard"/> <category name="calendarData"/> - </categoryList> - </action> In the message trace of SNOOPER, we see the clients action: CUccSubscriptionInfo::SetOpStatusForPresentity - Updating status for presentity sip:[email protected] to 0x80ef012d From here the client receive, if available the presence update. IM Sessions Interestingly, the IM can be seen similar with an email communication. Where a user is sending the IM via all involved servers. This explains, even if connection to a server is broken, audio/ video session (a peer-to-peer connection) stay established, but IM will be unavailable. The IM message is a one-way directed TLS connection from the sending client (in this illustration below) to his Edge server, via the Director, to Lync server and, the Lync server know the location of the target participant. IM Traffic (SIP) 5. IM replies in the opposite direction 4. IM is send to client (SIP/ TLS) 1. IM sent in SIP 2. Edge forwards IM to 3. Director Pool forwards connection secured with Director Pool Server (SIP/ IM to Frontend Pool (SIP/ TLS MTLS) MTLS) This explains why we sometime see an issue in communication, where user complain they can see the presence and be able singing IM to a target, but the target can’t reply. This is subject to an issue in the returning way. Either be port blockings or other network related issues. Audio/Video Session (Desktop/ Application - Sharing) In Lync and Skype for business, all audio and video related data will be exchanged in a peer-to-peer manner. But this is not valid for the session establishment. First the client send an INVITE (as we will see later in chapter: Analyzing real world call setup) from where the a/v path will be established directly between the two participants. In troubleshooting you have to analyze the both paths, first the session and afterwards the a/v path. The session establishment follows the path as described with IM. Other P2P connections are: Desktop Sharing and File Transfer, both secured with SRTP protocol IM Traffic (SIP) SRTP (SIP) 7. A/V session is established via P2P connection, secured with SRTP protocol 3. IM Session is send to 1. Initiate IM Session via Client (SIP/ TLS) Home Pool (SIP/ TLS) Bidirectional Channel 4. Client add A/V to the 2.IM Session is forward 5. Signaling is forwarded 6. Signaling is send to IM Session (Signaling) via Lync Pool A to second Pool (SIP/ to second Pool (SIP/ Lync Pool B Client (SIP/ TLS) SIP/TLS/MTLS) MTLS) MTLS) The path for a/v depends on the exchange of candidate pairs, you first have to analyze which candidates were send from both site and figure out the final candidates, the client want to establish along. (Also this process will be explained later in more detail) Conferencing A Conference is very similar to a normal SIP call, the main difference here is, that the user contact a MCU (Multipoint connection Unit) the conferencing server, which will handle all incoming and outgoing media streams. Here is a trace of an ad-hoc conference (“Meet now option”): Let us have a look into the process. We send an invite to the MCU service, which will be normally acknowledged and established as every other SIP call. The difference here is the conference ID which is submitted. The user setting up the conference will include the following application data: The conference key is the identifier for this conference (“yellow”) and we can see the C3P (CCCP) Microsoft conference protocol is used. Additionally we submit multiple other information, like the participant rule in this conference, here (“ATTENDEE”). This is normal, because if a user joint initially the conference, he should have the lowest rights. Content-Type: application/cccp+xml Content-Length: 964 - <request xmlns="urn:ietf:params:xml:ns:cccp" mscp="http://schemas.microsoft.com/rtc/2005/08/cccpextensions" C3PVersion="1" to="sip:[email protected];gruu;opaque=app:conf:focus:id:PTRL3DQ4" from="sip:thomas.poett@ domain-a.com" requestId="504251766368"> - <addUser> <conferenceKeys confEntity="sip:thomas.poett@ domain- a.com;gruu;opaque=app:conf:focus:id:PTRL3DQ4"/> - <user ci="urn:ietf:params:xml:ns:conference-info" entity="sip:thomas.poett@ domain-a.com"> - <roles> <entry>attendee</entry> - </roles> - <endpoint entity="{F0228E9E-3B8C-445B-A2AC-343A9FE7735B}" msci="http://schemas.microsoft.com/rtc/2005/08/confinfoextensions"> - <clientInfo> <separator cis="urn:ietf:params:xml:ns:conference-info-separator"/> <lobby-capable msci2="http://schemas.microsoft.com/rtc/2008/12/confinfoextensions">true</lobby-capable> - </clientInfo> - </endpoint> - </user> - </addUser> Later in the SIP/200 INVITE DIALOG CREATED, we see the escalation to the (“PRESENTER”) rule. - <user entity="sip:[email protected]"> - <roles> <entry>presenter</entry> - </roles> The most interesting message is the last INFO message from 8:57:08.501, here the conferencing service acknowledged all important information regarding web conferencing to the client. This is a huge XML message included. The RULE ENTRY is where those information are provided. NOTE: For troubleshooting it has two aspects, the client side as show in the illustration above and the server side. The conference here took place in between 08:57:08 and 08:57:35. On the client side no further information are provided what was happened in this conference. There you need to start MCU logging on the Lync/ Skype for Business server itself. When a call is escalated into a conference It its necessary to explain further more about the peep-o-peer call and a conference. As we remember, the audio/ video is always peer-to-peer, this include also desktop and application sharing, because of those data is similar to video. But what is with other services as POOLS, WHITE BOARD, or Q&A? This are service subject to conferences! Remember: Every time a user in a call using one of those services, the call will be first escalated into a conference! That’s just not all. There is one more very specific conference service, the Power Point presentation. If we are going to setup a webcast, or you want to share the presentation upfront with conference participants, you can upload files into a conference. The Power Point is here special and will be uploaded to the conference directory, like all other files too. But from here it will be rendered during a presentation into HTML 5.0 document. The Lync Client as well the Web Conferencing are HTML 5.0 aware and can present the Power Point data directly from the Office Web Application Server (WAS or WAC). Troubleshooting this server component is a chapter on his own. Lync Call Setup Call Setup over EDGE Server (General) Next we want to analyze a complex flow from two different Lync. One client sitting inside its LAN and the other remote (Internet). They need to communicate over the Edge Server. USER A (Internet) Edge Server Pool USER B (LAN) A/V Edge service SIP INVITE SIP INVITE SIP INVITE SIP 100 TRYING SIP 100 TRYING SIP 100 TRYING SIP 180 RINGING SIP 180 RINGING SIP 180 RINGING SIP 183 SESSION PROGRESS SIP 183 SESSION PROGRESS SIP 183 SESSION PROGRESS PRACK PRACK PRACK Candidate testing Candidate testing SIP 200 OK (PRACK) SIP 200 OK (PRACK) SIP 200 OK (PRACK) SIP 200 OK (INVITE) SIP 200 OK (INVITE) SIP 200 OK (INVITE) SIP ACK SIP ACK SIP ACK SIP INVITE SIP INVITE SIP INVITE SIP 100 TRYING SIP 100 TRYING SIP 100 TRYING SIP 200 OK SIP 200 OK SIP 200 OK SIP ACK SIP ACK SIP ACK SESSION IS ESTABLISHED Media Session Media Session While the client continue negotiating their best possible IP path, the media is establish during the negotiation process already. In the next chapter, we are going to have look into a real-world call establishment, where two federated partners setting up a call. Therefore also two Edge server are involved. In comparison to the call flow diagram from above, we will simply have one more additional hub in this scenario. Analyzing real world call setup I provide an example, where one client is external (remote) belonging to domain-a.com and the second client is internal (LAN) belonging to domain-b.com. This is the setup of an Audio call. Unnecessary line are removed. From the second communication extract onwards. [email protected] INVITE’s [email protected]. The direction provided is seen from the initiating client, meaning “OUTGOING” the client is sending a SIP command. “INCOMING” the client receives a SIP command. INVITE the USER (OUTGOING) Starting with the INVITE, where User A invites User B into a voice call. INVITE sip:[email protected] SIP/2.0 (whom to invite) Via: SIP/2.0/TLS 192.168.0.16:54763 (from where, the client IP address) Max-Forwards: 70 From: <sip:[email protected]>;tag=c013887c44;epid=0639570a7f (the user initiating the invite) To: <sip:[email protected]> (the whom to invite as target) Call-ID: ab5a007ca2124e95a227f1c82f58cff9 (our call identifier, if you search for a dedicated session search based on this ID in SNOOPER) CSeq: 1 INVITE (the SIP message sequence: 1st Invite) Contact: <sip:[email protected];opaque=user:epid:2b02BQuhtlW-2_O2vbNPYAAA;gruu> User-Agent: UCCAPI/15.0.4667.1000 OC/15.0.4667.1000 (Microsoft Lync) (which client is in used) Supported: ms-dialog-route-set-update (this section describes the supported features within this call, e.g if early-media is possible) Supported: timer Supported: histinfo Supported: ms-safe-transfer Supported: ms-sender Supported: ms-early-media Supported: 100rel Supported: replaces Supported: ms-conf-invite Ms-Conversation-ID: AdAQcNCXWtgHtgCCTziAoXiNeIlHgg== ms-keep-alive: UAC;hop-hop=yes Allow: INVITE, BYE, ACK, CANCEL, INFO, UPDATE, REFER, NOTIFY, BENOTIFY, OPTIONS (which SIP commands are available in this session) ms-subnet: 192.168.0.0 (the inviting client is within the IP network) Accept-Language: en-US ms-endpoint-location-data: NetworkScope;ms-media-location-type=Internet (the client is connecting via the internet, non-internal client) P-Preferred-Identity: <sip:[email protected]>, <tel:+4989zzyy75xx> (the identity submits information which could be used from the target site) Proxy-Authorization: TLS-DSK qop="auth", realm="SIP Communications Service", opaque="9731FB93", targetname="internalFQDN.domain-a.internal", crand="fe334d12", cnum="1224", response="c262f61fccd9b7e7e915e9a4f5f8b0fb31bdcdd2" (Authentication realm) Content-Type: multipart/alternative;boundary="----=_NextPart_000_0171_01D0107A.BB7313C0" Content-Length: 5434 ------=_NextPart_000_0171_01D0107A.BB7313C0 Content-Type: application/sdp Content-Transfer-Encoding: 7bit Content-ID: <[email protected]> Content-Disposition: session; handling=optional; ms-proxy-2007fallback v=0 o=- 0 0 IN IP4 195.145.140.92 s=session c=IN IP4 195.145.140.92 b=CT:99980 t=0 0 m=audio 54712 RTP/AVP 114 104 9 112 111 0 8 116 115 103 97 13 118 101 (This “blue” section is referring to the connection possibilities, first for compatibility reasons, the “older” version for OCS) a=candidate:Q4qp+2B3Rejg21tBRcAGfwXRQxwFbPH03U7efzcCp9M 1 75+hnK4QFySwmPMqbWWhJg UDP 0.830 192.168.0.16 21722 a=candidate:Q4qp+2B3Rejg21tBRcAGfwXRQxwFbPH03U7efzcCp9M 2 75+hnK4QFySwmPMqbWWhJg UDP 0.830 192.168.0.16 21723 a=candidate:hxajgyQ3ey1c0Dwrnogo1as2lNIZdsPbHDVH7UJy7mw 1 Uqk/hGPBtIDASJArq1AR2Q TCP 0.110 195.145.140.92 52196 a=candidate:hxajgyQ3ey1c0Dwrnogo1as2lNIZdsPbHDVH7UJy7mw 2 Uqk/hGPBtIDASJArq1AR2Q TCP 0.110 195.145.140.92 52196 a=candidate:Y+hVJGzBEjj4wtencMChvT7yEI8r6nHIPnQXB9Zchyw 1 Zfwzh/lMlJ/3YmOnpiJRdQ UDP 0.410 195.145.140.92 54712 a=candidate:Y+hVJGzBEjj4wtencMChvT7yEI8r6nHIPnQXB9Zchyw 2 Zfwzh/lMlJ/3YmOnpiJRdQ UDP 0.410 195.145.140.92 53613 a=candidate:M8BU29mhcNhmUej1BczG4sPn3yIb+3Rd5zc+bk0uWpY 1 k4fI8epJYEGk6WOVy/xfjA TCP 0.250 178.26.121.167 22616 a=candidate:M8BU29mhcNhmUej1BczG4sPn3yIb+3Rd5zc+bk0uWpY 2 k4fI8epJYEGk6WOVy/xfjA TCP 0.250 178.26.121.167 22616 a=candidate:8+mHOK0Z2Ks8kPymJH6FYE5RuFGbKNB/5ktLSfSF3Ns 1 gnqS5G1KOT/WlJbrw5d/uA UDP 0.550 178.26.121.167 23804 a=candidate:8+mHOK0Z2Ks8kPymJH6FYE5RuFGbKNB/5ktLSfSF3Ns 2 gnqS5G1KOT/WlJbrw5d/uA UDP 0.550 178.26.121.167 23805 a=cryptoscale:1 client AES_CM_128_HMAC_SHA1_80 inline:qk2+Fe5CWZcKttfJVk8wShtvC0ixVWQvVx1tYgYg|2^31|1:1 a=crypto:2 AES_CM_128_HMAC_SHA1_80 inline:eanuVGjZ1PGHP9Eh17H9u5uAr36WW2pjXzrH0s7W|2^31|1:1 a=crypto:3 AES_CM_128_HMAC_SHA1_80 inline:39IvYCDOPzx/MwuZNKuN5RHY+RSA0xz2LWGkBok3|2^31 a=maxptime:200 a=rtcp:53613 a=rtpmap:114 x-msrta/16000 a=fmtp:114 bitrate=29000 a=rtpmap:104 SILK/16000 (SKYPE CODEC) a=fmtp:104 useinbandfec=1; usedtx=0 a=rtpmap:9 G722/8000 a=rtpmap:112 G7221/16000 a=fmtp:112 bitrate=24000 a=rtpmap:111 SIREN/16000 a=fmtp:111 bitrate=16000 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:116 AAL2-G726-32/8000 a=rtpmap:115 x-msrta/8000 a=fmtp:115 bitrate=11800 a=rtpmap:103 SILK/8000 a=fmtp:103 useinbandfec=1; usedtx=0 a=rtpmap:97 RED/8000 a=rtpmap:13 CN/8000 a=rtpmap:118 CN/16000 a=rtpmap:101 telephone-event/8000 a=fmtp:101 0-16 a=ptime:20 (This “red” section is referring to the possible codes with can be utilized) ------=_NextPart_000_0171_01D0107A.BB7313C0 Content-Type: application/sdp Content-Transfer-Encoding: 7bit Content-ID: <[email protected]> Content-Disposition: session; handling=optional v=0 o=- 0 1 IN IP4 195.145.140.92 s=session c=IN IP4 195.145.140.92 b=CT:99980 t=0 0 a=x-devicecaps:audio:send,recv;video:send,recv m=audio 57962 RTP/AVP 114 104 9 112 111 0 8 116 115 103 97 13 118 101 a=x-ssrc-range:862104576-862104576 a=rtcp-fb:* x-message app send:dsh recv:dsh a=rtcp-rsize a=label:main-audio a=x-source:main-audio a=ice-ufrag:ccwh a=ice-pwd:8X3nfjTthYjuA8vhf+z2FYqM (This “blue” section is referring to the connection possibilities, now for Lync and Skype for Business) a=candidate:1 1 UDP 2130706431 192.168.0.16 10668 typ host a=candidate:1 2 UDP 2130705918 192.168.0.16 10669 typ host a=x-candidate-ipv6:2 1 UDP 2130705919 fd00:26:5bcb:fb92:d812:2961:203d:afd3 15206 typ host a=x-candidate-ipv6:2 2 UDP 2130705406 fd00:26:5bcb:fb92:d812:2961:203d:afd3 15207 typ host a=x-candidate-ipv6:3 1 UDP 33553407 2001:0:5ef5:79fd:34ae:19fd:4de5:8658 23084 typ host a=x-candidate-ipv6:3 2 UDP 33552894 2001:0:5ef5:79fd:34ae:19fd:4de5:8658 23085 typ host a=candidate:4 1 TCP-PASS 174455295 195.145.140.92 54427 typ relay raddr 178.26.121.167 rport 11603 a=candidate:4 2 TCP-PASS 174454782 195.145.140.92 54427 typ relay raddr 178.26.121.167 rport 11603 a=candidate:5 1 UDP 184547327 195.145.140.92 57962 typ relay raddr 178.26.121.167 rport 6098 a=candidate:5 2 UDP 184546814 195.145.140.92 51825 typ relay raddr 178.26.121.167 rport 6099 a=candidate:6 1 UDP 1694234111 178.26.121.167 6098 typ srflx raddr 192.168.0.16 rport 6098 a=candidate:6 2 UDP 1694233598 178.26.121.167 6099 typ srflx raddr 192.168.0.16 rport 6099 a=candidate:7 1 TCP-ACT 174846975 195.145.140.92 54427 typ relay raddr 178.26.121.167 rport 11603 a=candidate:7 2 TCP-ACT 174846462 195.145.140.92 54427 typ relay raddr 178.26.121.167 rport 11603 a=candidate:8 1 TCP-ACT 1684795903 178.26.121.167 11603 typ srflx raddr 192.168.0.16 rport 11603 a=candidate:8 2 TCP-ACT 1684795390 178.26.121.167 11603 typ srflx raddr 192.168.0.16 rport 11603 a=cryptoscale:1 client AES_CM_128_HMAC_SHA1_80 inline:qk2+Fe5CWZcKttfJVk8wShtvC0ixVWQvVx1tYgYg|2^31|1:1 a=crypto:2 AES_CM_128_HMAC_SHA1_80 inline:eanuVGjZ1PGHP9Eh17H9u5uAr36WW2pjXzrH0s7W|2^31|1:1 a=crypto:3 AES_CM_128_HMAC_SHA1_80 inline:39IvYCDOPzx/MwuZNKuN5RHY+RSA0xz2LWGkBok3|2^31 a=maxptime:200 a=rtcp:51825 a=rtpmap:114 x-msrta/16000 a=fmtp:114 bitrate=29000 a=rtpmap:104 SILK/16000 a=fmtp:104 useinbandfec=1; usedtx=0 a=rtpmap:9 G722/8000 a=rtpmap:112 G7221/16000 a=fmtp:112 bitrate=24000 a=rtpmap:111 SIREN/16000 a=fmtp:111 bitrate=16000 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:116 AAL2-G726-32/8000 a=rtpmap:115 x-msrta/8000 a=fmtp:115 bitrate=11800 a=rtpmap:103 SILK/8000 a=fmtp:103 useinbandfec=1; usedtx=0 a=rtpmap:97 RED/8000 a=rtpmap:13 CN/8000 a=rtpmap:118 CN/16000 a=rtpmap:101 telephone-event/8000 a=fmtp:101 0-16 (This “red” section is referring to the possible codes with can be utilized) a=ptime:20 TRYING (INCOMIG) SIP/2.0 100 Trying (Response from the target that its processing the Invite) ms-user-logon-data: RemoteUser Authentication-Info: TLS-DSK qop="auth", opaque="9731FB93", srand="7A0933C0", snum="1229", rspauth="1ae94e2225388db0f72729407866c59710b3d463", targetname="internalFQDN.domain- a.internal", realm="SIP Communications Service", version=4 From: <sip:[email protected]>;tag=c013887c44;epid=0639570a7f To: <sip:[email protected]> Call-ID: ab5a007ca2124e95a227f1c82f58cff9 CSeq: 1 INVITE (message response from the 1st, initial invite) Via: SIP/2.0/TLS 192.168.0.16:54763;received=178.26.121.167;ms-received-port=54763;ms-received- cid=86B100 (Information about the path seeing the internal sender IP and the NAT IP on the Internet Connection) Server: http%3A%2F%2Fwww.microsoft.com%2FLCS%2FUdcAgent Content-Length: 0 SESSION PROGRESS (INCOMING) Here we are informed that the session is in progress. The target system is processing the session and will send more about the process soon. The CSeq is still indicating the dependency on the first INVITE. SIP/2.0 183 Session Progress ms-user-logon-data: RemoteUser Authentication-Info: TLS-DSK qop="auth", opaque="9731FB93", srand="F7BE52E1", snum="1231", rspauth="6d8fca262a42dc48169ef0142a2a2b910db30ba5", targetname="internalFQDN.domain- a.internal", realm="SIP Communications Service", version=4 Content-Length: 0 Via: SIP/2.0/TLS 192.168.0.16:54763;received=178.26.121.167;ms-received-port=54763;ms-received- cid=86B100 ( From: "Thomas Poett"<sip:[email protected]>;tag=c013887c44;epid=0639570a7f To: <sip:[email protected]> Call-ID: ab5a007ca2124e95a227f1c82f58cff9 CSeq: 1 INVITE Ms-Forking: Active Server: http%3A%2F%2Fwww.microsoft.com%2FLCS%2FDefaultRouting(Microsoft Lync Server 2013 5.0.8308.726) ms-edge-proxy-message-trust: ms-source-type=DirectPartner;ms-ep- fqdn=lyncedgepool.internFQDN.local;ms-source-network=federation;ms-source-verified- user=verified (interestingly we see the message is running through the Edge server) PROGRESS REPORT (INCOMING) -2 times (identically send) Still processing (send has to wait for next message) SIP/2.0 101 Progress Report ms-user-logon-data: RemoteUser Authentication-Info: TLS-DSK qop="auth", opaque="9731FB93", srand="1E5DE761", snum="1232", rspauth="e415d70ab0d015011336756cb385f8c0aa35e55e", targetname="internalFQDN.domain- a.internal", realm="SIP Communications Service", version=4 Content-Length: 0 Via: SIP/2.0/TLS 192.168.0.16:54763;received=178.26.121.167;ms-received-port=54763;ms-received- cid=86B100 From: "Thomas Poett"<sip:[email protected]>;tag=c013887c44;epid=0639570a7f To: <sip:[email protected]> Call-ID: ab5a007ca2124e95a227f1c82f58cff9 CSeq: 1 INVITE Server: InboundRouting/5.0.0.0 (send by target server) ms-edge-proxy-message-trust: ms-source-type=DirectPartner;ms-ep- fqdn=lyncedgepool.internFQDN.local;ms-source-network=federation;ms-source-verified- user=verified RINGING (INCOMING) – 4 times This is repeated until the user pickup, rejects or don’t answer (timeout). The Ringing response is given for each device connected on the target site. SIP/2.0 180 Ringing ms-user-logon-data: RemoteUser Via: SIP/2.0/TLS 192.168.0.16:54763;received=178.26.121.167;ms-received-port=54763;ms-received- cid=86B100 Authentication-Info: TLS-DSK qop="auth", opaque="9731FB93", srand="FA1B1315", snum="1234", rspauth="c74126037c499380d505ad2902f868320c3a6a95", targetname="internalFQDN.domain- a.internal", realm="SIP Communications Service", version=4 FROM: "Thomas Poett"<sip:[email protected]>;tag=c013887c44;epid=0639570a7f TO: "USER B"<sip:[email protected]>;tag=32653fb637;epid=212448855504 CSEQ: 1 INVITE CALL-ID: ab5a007ca2124e95a227f1c82f58cff9 CONTACT: <sip:userb@domain- b.com;opaque=user:epid:OWi6ihJpQlqfhlrZfAmZywAA;gruu>;text;audio;video;image;applicationshari ng CONTENT-LENGTH: 0 SUPPORTED: gruu-10 ALLOW: CANCEL (The ALLOW show the possible commands based on the Ringing, meaning how to terminate and more…) ALLOW: BYE ALLOW: UPDATE ALLOW: PRACK P-ASSERTED-IDENTITY: "USER B"<sip:[email protected]> SERVER: RTCC/5.0.0.0 UCWA/5.0.0.0 AndroidLync/4.0.0000.0000 (Nexus 7 Android 4.4.4) [As we saw in the initial screen shot, 4 RINGING response were received, this are the green marked devices User-Agent: UCCAPI/15.0.4667.1000 OC/15.0.4667.1000 (Microsoft Lync) SERVER: RTCC/5.0.0.0 UCWA/5.0.0.0 AndroidLync/5.4.1106.0 (GXV3275 Android 4.2.2) User-Agent: CPE/4.0.7577.4451 OCPhone/4.0.7577.4451 (Microsoft Lync Phone Edition)] ms-edge-proxy-message-trust: ms-source-type=DirectPartner;ms-ep- fqdn=lyncedgepool.internFQDN.local;ms-source-network=federation;ms-source-verified- user=verified PROGRESS REPORT (INCOMING) (The call was now take by USER B (callee), we need another progress report here stating this). We also need to receive the SDP here that early-media can be established with this candiates). This is send from the device taking the call to connect with audio. In this trace from a Lync native client. SIP/2.0 183 Session Progress ms-user-logon-data: RemoteUser Authentication-Info: TLS-DSK qop="auth", opaque="9731FB93", srand="CD84738D", snum="1238", rspauth="57062b086f1b35c4b848bc42fb28b33897f6963e", targetname="internalFQDN.domain- a.internal", realm="SIP Communications Service", version=4 Via: SIP/2.0/TLS 192.168.0.16:54763;received=178.26.121.167;ms-received-port=54763;ms-received- cid=86B100 From: "Thomas Poett"<sip:[email protected]>;tag=c013887c44;epid=0639570a7f To: <sip:[email protected]>;epid=a606e73c89;tag=01c33dbedc Call-ID: ab5a007ca2124e95a227f1c82f58cff9 CSeq: 1 INVITE (we are still in progress receiving information based on the first INVITE) [we first identify the target sites route and involved servers and its CANDIDATES HERE establishing early-media] Record-Route: <sip:lync2013.hq.domain-b.com:5061;transport=tls;opaque=state:T;lr> Record-Route: <sip:ocs.domain-b.com:5061;transport=tls;epid=a606e73c89;lr;ms-key-info=.......;ms- route-sig=bkNIsAXKnsiRmbGYKjO4lps9o6nw6ySSsyRfgI3kTNTO9ouKskkfZA1AAA> Record-Route: <sip:lyncedgepool.internFQDN.local:5061;transport=tls;lr> Record-Route: <sip:lyncpool1.internFQDN.local:5061;transport=tls;ms-fe=internalFQDN.domain- a.internal;opaque=state:F;lr;received=10.35.3.27;ms-received-cid=78E300> Record-Route: <sip:sip.domain-a.com:5061;transport=tls;opaque=state:Ci.R86b100;lr;ms-route- sig=fgFbc2J6H1DsQORB0QtvkWygilYrA-0Ccj13Tr9fH6LkbccFlf6mOs6wAA> Contact: <sip:[email protected];opaque=user:epid:LLsGlnCjYlGGrdmzIoEFUwAA;gruu> User-Agent: UCCAPI/15.0.4667.1000 OC/15.0.4667.1000 (Microsoft Lync) Require: 100rel RSeq: 1 (Receive CSEQ, now the target site requests a response) ms-endpoint-location-data: NetworkScope;ms-media-location-type=Intranet Content-Type: application/sdp Content-Length: 1894 ms-edge-proxy-message-trust: ms-source-type=DirectPartner;ms-ep- fqdn=lyncedgepool.internFQDN.local;ms-source-network=federation;ms-source-verified- user=verified v=0 o=- 0 0 IN IP4 10.6.0.5 s=session c=IN IP4 10.6.0.5 b=CT:99980 t=0 0 a=x-devicecaps:audio:send,recv;video:send,recv m=audio 9450 RTP/SAVP 114 9 112 111 0 8 116 115 97 13 118 101 a=x-ssrc-range:76626436-76626436 a=rtcp-fb:* x-message app send:dsh recv:dsh a=rtcp-rsize a=label:main-audio a=x-source:main-audio a=ice-ufrag:MNHU a=ice-pwd:5kvPXMuiOnrb7F2p4DG15DQm (all possible candidates from the target system are submitted) a=candidate:1 1 UDP 2130706431 10.6.0.5 9450 typ host a=candidate:1 2 UDP 2130705918 10.6.0.5 9451 typ host a=candidate:2 1 UDP 2130705919 192.168.198.1 19232 typ host a=candidate:2 2 UDP 2130705406 192.168.198.1 19233 typ host a=candidate:3 1 TCP-PASS 174455807 188.111.10.69 58236 typ relay raddr 10.6.0.5 rport 31472 a=candidate:3 2 TCP-PASS 174455294 188.111.10.69 58236 typ relay raddr 10.6.0.5 rport 31472 a=candidate:4 1 UDP 184547839 188.111.10.69 56186 typ relay raddr 10.6.0.5 rport 29624 a=candidate:4 2 UDP 184547326 188.111.10.69 52054 typ relay raddr 10.6.0.5 rport 29625 a=candidate:5 1 TCP-ACT 174847999 188.111.10.69 58236 typ relay raddr 10.6.0.5 rport 31472 a=candidate:5 2 TCP-ACT 174847486 188.111.10.69 58236 typ relay raddr 10.6.0.5 rport 31472 a=candidate:6 1 TCP-ACT 1684796927 10.6.0.5 31472 typ srflx raddr 10.6.0.5 rport 31472 a=candidate:6 2 TCP-ACT 1684796414 10.6.0.5 31472 typ srflx raddr 10.6.0.5 rport 31472 a=crypto:2 AES_CM_128_HMAC_SHA1_80 inline:qoMfVcC0DhPKn3Fs7/9n2BL2iM0P4XSgpxeXR874|2^31|1:1 a=maxptime:200 a=rtpmap:114 x-msrta/16000 a=fmtp:114 bitrate=29000 a=rtpmap:9 G722/8000 a=rtpmap:112 G7221/16000 a=fmtp:112 bitrate=24000 a=rtpmap:111 SIREN/16000 a=fmtp:111 bitrate=16000 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:116 AAL2-G726-32/8000 a=rtpmap:115 x-msrta/8000 a=fmtp:115 bitrate=11800 a=rtpmap:97 RED/8000 a=rtpmap:13 CN/8000 a=rtpmap:118 CN/16000 a=rtpmap:101 telephone-event/8000 a=fmtp:101 0-16 a=ptime:20 PRACK (OUTGOING) Reliability of “Provisional Responses” in the Session Initiation Protocol (PRACK RFC3262). Two possible responses exit: provisional and final. PRACK is the provisional response. We signalize: We are ready to connect. PRACK sip:[email protected];opaque=user:epid:LLsGlnCjYlGGrdmzIoEFUwAA;gruu SIP/2.0 Via: SIP/2.0/TLS 192.168.0.16:54763 Max-Forwards: 70 From: <sip:[email protected]>;tag=c013887c44;epid=0639570a7f To: <sip:[email protected]>;epid=a606e73c89;tag=01c33dbedc Call-ID: ab5a007ca2124e95a227f1c82f58cff9 CSeq: 2 PRACK (CSEQ is increased to 2, we continue with the next process establishing the call) Route: <sip:sip.domain-a.com:5061;transport=tls;opaque=state:Ci.R86b100;lr;ms-route- sig=fgFbc2J6H1DsQORB0QtvkWygilYrA-0Ccj13Tr9fH6LkbccFlf6mOs6wAA> Route: <sip:lyncpool1.internFQDN.local:5061;transport=tls;ms-fe=internalFQDN.domain- a.internal;opaque=state:F;lr;received=10.35.3.27;ms-received-cid=78E300> Route: <sip:lyncedgepool.internFQDN.local:5061;transport=tls;lr> Route: <sip:ocs.domain-b.com:5061;transport=tls;epid=a606e73c89;lr;ms-key- info=AAEAAcUcKOgG4q8UZBDQAVcU1Pt2m_6ekgNlaaYrCW5qBdhWPxt97pJbLWjJqjzzJFfUkk6MJ2eS yIm77AmUE76vtBbPqEm5zKxc8R9DxeBx2FZfM49Y16vsDldXQl4uDazo3BWXJDggmtpWiOWkYwhmW adsT813jJ9JhAIxcPTzltiNbNZ9b0_iffEi6oI6_HlftQCLqKb-n8tq1gIsAtSYw8-o2vGxrswJ8HeIFn6z0TFE- Sd_F76aV8DYmi-NkRPcwknKnBMwDhNKQ2pl2vZ61B9oFlcy9xFQOdLZujQUyZ27T06xfFqzz3H- bPyd92DSPFmvjfcvGx6WGqQHI4FIkz2pe9FIKXi8ZVVJhS6EE65YtcKxvu5LfCxSN5P4ZePtClNSdlyV2FLyOt WYXS72c1xHNM5AT53D2KJjwzRURJ1v--1CLkVuiFfCPe8GSPVbqTLmdNLlmpqtF05UiEa- 9coZDWDdYx41iGUNxfhykgSt2iFoTyjdShLxTg6BbAS5zuf5BY_VUuJwKjx9b0- ecWhYzTKK_JaQDim4WGPFqRx8PAcx4GZtv6OQMAwYA8A5ZeXLQWa0DL_PFq4BVF5HfR_jfyQciEGjV Gsc5BXT0CdprYmXNjSyYLzEG4WzCm4jpdX_SnInhJhZ1SZ8kv9n91oV7O7MVwGyBN6ALBbUVCUugkj5 Nfn0qzv_UKDM;ms-route- sig=bkNIsAXKnsiRmbGYKjO4lps9o6nw6ySSsyRfgI3kTNTO9ouKskkfZA1AAA> Route: <sip:lync2013.hq.domain-b.com:5061;transport=tls;opaque=state:T;lr> User-Agent: UCCAPI/15.0.4667.1000 RAck: 1 1 INVITE (a PAck based on our INVITE) Proxy-Authorization: TLS-DSK qop="auth", realm="SIP Communications Service", opaque="9731FB93", targetname="internalFQDN.domain-a.internal", crand="ae68c543", cnum="1226", response="8913674f898d1f2c357350d9e8e9c348d7abc36b" Content-Length: 0 OK (INCOMING) The PRACK was acknowledged from the SIP/2.0 200 OK ms-user-logon-data: RemoteUser Authentication-Info: TLS-DSK qop="auth", opaque="9731FB93", srand="0B8D6EC0", snum="1239", rspauth="971556595d08e9e1977484380009bfb0457afeb3", targetname="internalFQDN.domain- a.internal", realm="SIP Communications Service", version=4 Via: SIP/2.0/TLS 192.168.0.16:54763;received=178.26.121.167;ms-received-port=54763;ms-received- cid=86B100 From: <sip:[email protected]>;tag=c013887c44;epid=0639570a7f To: <sip:[email protected]>;epid=a606e73c89;tag=01c33dbedc Call-ID: ab5a007ca2124e95a227f1c82f58cff9 CSeq: 2 PRACK User-Agent: UCCAPI/15.0.4667.1000 OC/15.0.4667.1000 (Microsoft Lync) Content-Length: 0 ms-edge-proxy-message-trust: ms-source-type=DirectPartner;ms-ep- fqdn=lyncedgepool.internFQDN.local;ms-source-network=federation;ms-source-verified- user=verified SESSION PROGRESS (INCOMING) The target is signalizing it is processing the still on the first INVITE. Even it has received the PRACK and answered for provisional progress. It was also repeating the candidates once more for later use. This command 183 Session Progress is the responsible trigger for CALLEE’S device signalizing the incoming call on his/ her device! SIP/2.0 183 Session Progress ms-user-logon-data: RemoteUser Authentication-Info: TLS-DSK qop="auth", opaque="9731FB93", srand="2F18494A", snum="1240", rspauth="a62e2a1927663ce245b8992e1283c26aaa1716ea", targetname="internalFQDN.domain- a.internal", realm="SIP Communications Service", version=4 Via: SIP/2.0/TLS 192.168.0.16:54763;received=178.26.121.167;ms-received-port=54763;ms-received- cid=86B100 From: "Thomas Poett"<sip:[email protected]>;tag=c013887c44;epid=0639570a7f To: <sip:[email protected]>;epid=5385571cf9;tag=1ce13477ae Call-ID: ab5a007ca2124e95a227f1c82f58cff9 CSeq: 1 INVITE Record-Route: <sip:lync2013.hq.domain-b.com:5061;transport=tls;opaque=state:T;lr> Record-Route: <sip:ocs.domain-b.com:5061;transport=tls;epid=5385571cf9;lr;ms-key- info=AAEAAcUcKOgG4q8UZBDQAVcU1Pt2m_6ekgNlaaYrCW5qBdhWPxt97pJbLWjJqjzzJFfUkk6MJ2eS yIm77AmUE76vtBbPqEm5zKxc8R9DxeBx2FZfM49Y16vsDldXQl4uDazo3BWXJDggmtpWiOWkYwhmW adsT813jJ9JhAIxcPTzltiNbNZ9b0_iffEi6oI6_HlftQCLqKb-n8tq1gIsAtSYw8-o2vGxrswJ8HeIFn6z0TFE- Sd_F76aV8DYmi-NkRPcwknKnBMwDhNKQ2pl2vZ61B9oFlcy9xFQOdLZujQUyZ27T06xfFqzz3H- bPyd92DSPFmvjfcvGx6WGqQHI4FIkz2pe9FIKXi8ZVVJhS6EE65YtcKxvu5LfCxSN5P4ZePtClNSdlyV2FLyOt WYXS72c1xHNM5AT53D2KJjwzRURJ1v--1CLkVuiFfCPe8GSPVbqTLmdNLlmpqtF05UiEa- 9coZDWDdYx41iGUNxfhykgSt2iFoTyjdShLxTg6BbAS5zuf5BY_VUuJwKjx9b0- ecWhYzTKK_JaQDim4WGPFqRx8PAcx4GZtv6OQMAwYA8A5ZeXLQWa0DL_PFq4BVF5HfR_jfyQciEGjV Gsc5BXT0CdprYmXNjSyYLzEG4WzCm4jpdX_SnInhJhZ1SZ8kv9n91oV7O7MVwGyBN6ALBbUVCUugkj5 Nfn0qzv_UKDM;ms-route- sig=bk1thRQ_ihhCNLmtXzZOeqNXyFx22uhJJkDX_9qM1XZNBouKskkfZA1AAA> Record-Route: <sip:lyncedgepool.internFQDN.local:5061;transport=tls;lr> Record-Route: <sip:lyncpool1.internFQDN.local:5061;transport=tls;ms-fe=internalFQDN.domain- a.internal;opaque=state:F;lr;received=10.35.3.27;ms-received-cid=78E300> Record-Route: <sip:sip.domain-a.com:5061;transport=tls;opaque=state:Ci.R86b100;lr;ms-route- sig=fgH21o9hIUtQHEH7aVOkn_arBus-dKgtT5S5hpnFuJujrccFlf6mOs6wAA> Contact: <sip:[email protected];opaque=user:epid:uakLNJvpLFu9MOSL-0K82wAA;gruu> User-Agent: CPE/4.0.7577.4451 OCPhone/4.0.7577.4451 (Microsoft Lync Phone Edition) Require: 100rel RSeq: 1 ms-endpoint-location-data: NetworkScope;ms-media-location-type=Intranet Content-Type: application/sdp Content-Length: 1491 ms-edge-proxy-message-trust: ms-source-type=DirectPartner;ms-ep- fqdn=lyncedgepool.internFQDN.local;ms-source-network=federation;ms-source-verified- user=verified v=0 o=- 0 0 IN IP4 188.111.10.69 s=session c=IN IP4 188.111.10.69 b=CT:99980 t=0 0 m=audio 53534 RTP/SAVP 114 9 111 0 8 115 97 13 118 101 (The Answer with the opposite possible candidate pairs. Here answer with the candidate matching the Lync/ Skype for Business version only) a=ice-ufrag:P7RK a=ice-pwd:IPNjQORYMHFIXYSC4FMKg9j1 a=candidate:1 1 UDP 2130706431 10.6.3.6 25526 typ host a=candidate:1 2 UDP 2130705918 10.6.3.6 25527 typ host a=candidate:2 1 TCP-PASS 6556159 188.111.10.69 58180 typ relay raddr 10.6.3.6 rport 7680 a=candidate:2 2 TCP-PASS 6556158 188.111.10.69 58180 typ relay raddr 10.6.3.6 rport 7680 a=candidate:3 1 UDP 16648703 188.111.10.69 53534 typ relay raddr 10.6.3.6 rport 29434 a=candidate:3 2 UDP 16648702 188.111.10.69 51218 typ relay raddr 10.6.3.6 rport 29435 a=candidate:4 1 TCP-ACT 7076863 188.111.10.69 58180 typ relay raddr 10.6.3.6 rport 7680 a=candidate:4 2 TCP-ACT 7076350 188.111.10.69 58180 typ relay raddr 10.6.3.6 rport 7680 a=candidate:5 1 TCP-ACT 1684797951 10.6.3.6 7680 typ srflx raddr 10.6.3.6 rport 7680 a=candidate:5 2 TCP-ACT 1684797438 10.6.3.6 7680 typ srflx raddr 10.6.3.6 rport 7680 a=crypto:2 AES_CM_128_HMAC_SHA1_80 inline:zdjwrlJGdYxjeSS/gEeDvYDQT+9mc1F0yM6WVN1d|2^31|1:1 a=maxptime:200 a=rtcp:51218 a=rtpmap:114 x-msrta/16000 a=fmtp:114 bitrate=29000 a=rtpmap:9 G722/8000 a=rtpmap:111 SIREN/16000 a=fmtp:111 bitrate=16000 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:115 x-msrta/8000 a=fmtp:115 bitrate=11800 a=rtpmap:97 RED/8000 a=rtpmap:13 CN/8000 a=rtpmap:118 CN/16000 a=rtpmap:101 telephone-event/8000 a=fmtp:101 0-16 PRACK (OUTGOING) PRACK sip:[email protected];opaque=user:epid:uakLNJvpLFu9MOSL-0K82wAA;gruu SIP/2.0 Via: SIP/2.0/TLS 192.168.0.16:54763 Max-Forwards: 70 From: <sip:[email protected]>;tag=c013887c44;epid=0639570a7f To: <sip:[email protected]>;epid=5385571cf9;tag=1ce13477ae Call-ID: ab5a007ca2124e95a227f1c82f58cff9 CSeq: 2 PRACK Route: <sip:sip.domain-a.com:5061;transport=tls;opaque=state:Ci.R86b100;lr;ms-route- sig=fgH21o9hIUtQHEH7aVOkn_arBus-dKgtT5S5hpnFuJujrccFlf6mOs6wAA> Route: <sip:lyncpool1.internFQDN.local:5061;transport=tls;ms-fe=internalFQDN.domain- a.internal;opaque=state:F;lr;received=10.35.3.27;ms-received-cid=78E300> Route: <sip:lyncedgepool.internFQDN.local:5061;transport=tls;lr> Route: <sip:ocs.domain-b.com:5061;transport=tls;epid=5385571cf9;lr;ms-key- info=AAEAAcUcKOgG4q8UZBDQAVcU1Pt2m_6ekgNlaaYrCW5qBdhWPxt97pJbLWjJqjzzJFfUkk6MJ2eS yIm77AmUE76vtBbPqEm5zKxc8R9DxeBx2FZfM49Y16vsDldXQl4uDazo3BWXJDggmtpWiOWkYwhmW adsT813jJ9JhAIxcPTzltiNbNZ9b0_iffEi6oI6_HlftQCLqKb-n8tq1gIsAtSYw8-o2vGxrswJ8HeIFn6z0TFE- Sd_F76aV8DYmi-NkRPcwknKnBMwDhNKQ2pl2vZ61B9oFlcy9xFQOdLZujQUyZ27T06xfFqzz3H- bPyd92DSPFmvjfcvGx6WGqQHI4FIkz2pe9FIKXi8ZVVJhS6EE65YtcKxvu5LfCxSN5P4ZePtClNSdlyV2FLyOt WYXS72c1xHNM5AT53D2KJjwzRURJ1v--1CLkVuiFfCPe8GSPVbqTLmdNLlmpqtF05UiEa- 9coZDWDdYx41iGUNxfhykgSt2iFoTyjdShLxTg6BbAS5zuf5BY_VUuJwKjx9b0- ecWhYzTKK_JaQDim4WGPFqRx8PAcx4GZtv6OQMAwYA8A5ZeXLQWa0DL_PFq4BVF5HfR_jfyQciEGjV Gsc5BXT0CdprYmXNjSyYLzEG4WzCm4jpdX_SnInhJhZ1SZ8kv9n91oV7O7MVwGyBN6ALBbUVCUugkj5 Nfn0qzv_UKDM;ms-route- sig=bk1thRQ_ihhCNLmtXzZOeqNXyFx22uhJJkDX_9qM1XZNBouKskkfZA1AAA> Route: <sip:lync2013.hq.domain-b.com:5061;transport=tls;opaque=state:T;lr> User-Agent: UCCAPI/15.0.4667.1000 RAck: 1 1 INVITE Proxy-Authorization: TLS-DSK qop="auth", realm="SIP Communications Service", opaque="9731FB93", targetname="internalFQDN.domain-a.internal", crand="5ccff29a", cnum="1227", response="5848d72e4f8922c35d4731ed3b8baa03cefc576d" Content-Length: 0 PRE-CALL with EARLY-MEDIA ESTABLISHED ! You can listen to Audio from here… OK (INCOMING) Lync Phone Edition reports OK only. (Because the Desktop client picked up the call, see next Ok) – We can’t see the target client IP, since the OK was send from Lync server) SIP/2.0 200 OK ms-user-logon-data: RemoteUser Authentication-Info: TLS-DSK qop="auth", opaque="9731FB93", srand="E0730742", snum="1241", rspauth="1052a2e1c8fef69041775164c0e357aeb866b452", targetname="internalFQDN.domain- a.internal", realm="SIP Communications Service", version=4 Via: SIP/2.0/TLS 192.168.0.16:54763;received=178.26.121.167;ms-received-port=54763;ms-received- cid=86B100 From: <sip:[email protected]>;tag=c013887c44;epid=0639570a7f To: <sip:[email protected]>;epid=5385571cf9;tag=1ce13477ae Call-ID: ab5a007ca2124e95a227f1c82f58cff9 CSeq: 2 PRACK User-Agent: CPE/4.0.7577.4451 OCPhone/4.0.7577.4451 (Microsoft Lync Phone Edition) Content-Length: 0 ms-edge-proxy-message-trust: ms-source-type=DirectPartner;ms-ep- fqdn=lyncedgepool.internFQDN.local;ms-source-network=federation;ms-source-verified- user=verified OK (INCOMING) Same as above 200 OK. Here this is the client who picked up the call finally. (The “blue” highlighted ms-client-diagnostics provides the path, ports and IPs chosen/ involved in the INCOMING call) Use this information for troubleshooting if a call can’t be connected. SIP/2.0 200 OK ms-user-logon-data: RemoteUser Authentication-Info: TLS-DSK qop="auth", opaque="9731FB93", srand="8461DC50", snum="1243", rspauth="1af8acd0ed86a6ac7dc44718f1c166e326275718", targetname="internalFQDN.domain- a.internal", realm="SIP Communications Service", version=4 Via: SIP/2.0/TLS 192.168.0.16:54763;received=178.26.121.167;ms-received-port=54763;ms-received- cid=86B100 Content-Length: 1894 P-Asserted-Identity: <sip:[email protected]>, <tel:+493328455946;ext=946> From: "Thomas Poett"<sip:[email protected]>;tag=c013887c44;epid=0639570a7f To: <sip:[email protected]>;epid=a606e73c89;tag=01c33dbedc Call-ID: ab5a007ca2124e95a227f1c82f58cff9 CSeq: 1 INVITE Record-Route: <sip:lync2013.hq.domain-b.com:5061;transport=tls;opaque=state:T;lr> Record-Route: <sip:ocs.domain-b.com:5061;transport=tls;epid=a606e73c89;lr;ms-key- info=AAEAAcUcKOgG4q8UZBDQAVcU1Pt2m_6ekgNlaaYrCW5qBdhWPxt97pJbLWjJqjzzJFfUkk6MJ2eS yIm77AmUE76vtBbPqEm5zKxc8R9DxeBx2FZfM49Y16vsDldXQl4uDazo3BWXJDggmtpWiOWkYwhmW adsT813jJ9JhAIxcPTzltiNbNZ9b0_iffEi6oI6_HlftQCLqKb-n8tq1gIsAtSYw8-o2vGxrswJ8HeIFn6z0TFE- Sd_F76aV8DYmi-NkRPcwknKnBMwDhNKQ2pl2vZ61B9oFlcy9xFQOdLZujQUyZ27T06xfFqzz3H- bPyd92DSPFmvjfcvGx6WGqQHI4FIkz2pe9FIKXi8ZVVJhS6EE65YtcKxvu5LfCxSN5P4ZePtClNSdlyV2FLyOt WYXS72c1xHNM5AT53D2KJjwzRURJ1v--1CLkVuiFfCPe8GSPVbqTLmdNLlmpqtF05UiEa- 9coZDWDdYx41iGUNxfhykgSt2iFoTyjdShLxTg6BbAS5zuf5BY_VUuJwKjx9b0- ecWhYzTKK_JaQDim4WGPFqRx8PAcx4GZtv6OQMAwYA8A5ZeXLQWa0DL_PFq4BVF5HfR_jfyQciEGjV Gsc5BXT0CdprYmXNjSyYLzEG4WzCm4jpdX_SnInhJhZ1SZ8kv9n91oV7O7MVwGyBN6ALBbUVCUugkj5 Nfn0qzv_UKDM;ms-route- sig=bkNIsAXKnsiRmbGYKjO4lps9o6nw6ySSsyRfgI3kTNTO9ouKskkfZA1AAA> Record-Route: <sip:lyncedgepool.internFQDN.local:5061;transport=tls;lr> Record-Route: <sip:lyncpool1.internFQDN.local:5061;transport=tls;ms-fe=internalFQDN.domain- a.internal;opaque=state:F;lr;received=10.35.3.27;ms-received-cid=78E300> Record-Route: <sip:sip.domain-a.com:5061;transport=tls;opaque=state:Ci.R86b100;lr;ms-route- sig=fgFbc2J6H1DsQORB0QtvkWygilYrA-0Ccj13Tr9fH6LkbccFlf6mOs6wAA> Contact: <sip:[email protected];opaque=user:epid:LLsGlnCjYlGGrdmzIoEFUwAA;gruu> User-Agent: UCCAPI/15.0.4667.1000 OC/15.0.4667.1000 (Microsoft Lync) Supported: histinfo Supported: ms-safe-transfer Supported: ms-dialog-route-set-update Supported: ms-bypass Supported: replaces Allow: INVITE, BYE, ACK, CANCEL, INFO, UPDATE, REFER, NOTIFY, BENOTIFY, OPTIONS Session-Expires: 720;refresher=uac Ms-Accepted-Content-ID: <[email protected]> ms-client-diagnostics: 51007;reason="Callee media connectivity diagnosis info";UserType="Callee";MediaType="audio";ICEWarn="0x0";LocalSite="10.6.0.5:9450";LocalMR="18 8.111.10.69:56186";RemoteSite="178.26.121.167:11603";RemoteMR="195.145.140.92:54427";Port Range="1025:65000";LocalMRTCPPort="58236";RemoteMRTCPPort="54427";LocalLocation="2";Rem oteLocation="1";FederationType="1";NetworkName="hq.domain- b";Interfaces="0x2";BaseInterface="0x2";BaseAddress="10.6.0.5:29624;MrDnsU="lyncedge2013.hq. domain-b.com";MrResU="0" ms-endpoint-location-data: NetworkScope;ms-media-location-type=Intranet Content-Type: application/sdp ms-edge-proxy-message-trust: ms-source-type=DirectPartner;ms-ep- fqdn=lyncedgepool.internFQDN.local;ms-source-network=federation;ms-source-verified- user=verified v=0 o=- 0 1 IN IP4 10.6.0.5 s=session c=IN IP4 10.6.0.5 b=CT:99980 t=0 0 a=x-devicecaps:audio:send,recv;video:send,recv m=audio 9450 RTP/SAVP 114 9 112 111 0 8 116 115 97 13 118 101 a=x-ssrc-range:76626436-76626436 a=rtcp-fb:* x-message app send:dsh recv:dsh a=rtcp-rsize a=label:main-audio a=x-source:main-audio (Acknowledgement of ICS candiatets) a=ice-ufrag:MNHU a=ice-pwd:5kvPXMuiOnrb7F2p4DG15DQm a=candidate:1 1 UDP 2130706431 10.6.0.5 9450 typ host a=candidate:1 2 UDP 2130705918 10.6.0.5 9451 typ host a=candidate:2 1 UDP 2130705919 192.168.198.1 19232 typ host a=candidate:2 2 UDP 2130705406 192.168.198.1 19233 typ host a=candidate:3 1 TCP-PASS 174455807 188.111.10.69 58236 typ relay raddr 10.6.0.5 rport 31472 a=candidate:3 2 TCP-PASS 174455294 188.111.10.69 58236 typ relay raddr 10.6.0.5 rport 31472 a=candidate:4 1 UDP 184547839 188.111.10.69 56186 typ relay raddr 10.6.0.5 rport 29624 a=candidate:4 2 UDP 184547326 188.111.10.69 52054 typ relay raddr 10.6.0.5 rport 29625 a=candidate:5 1 TCP-ACT 174847999 188.111.10.69 58236 typ relay raddr 10.6.0.5 rport 31472 a=candidate:5 2 TCP-ACT 174847486 188.111.10.69 58236 typ relay raddr 10.6.0.5 rport 31472 a=candidate:6 1 TCP-ACT 1684796927 10.6.0.5 31472 typ srflx raddr 10.6.0.5 rport 31472 a=candidate:6 2 TCP-ACT 1684796414 10.6.0.5 31472 typ srflx raddr 10.6.0.5 rport 31472 a=crypto:2 AES_CM_128_HMAC_SHA1_80 inline:qoMfVcC0DhPKn3Fs7/9n2BL2iM0P4XSgpxeXR874|2^31|1:1 a=maxptime:200 a=rtpmap:114 x-msrta/16000 a=fmtp:114 bitrate=29000 a=rtpmap:9 G722/8000 a=rtpmap:112 G7221/16000 a=fmtp:112 bitrate=24000 a=rtpmap:111 SIREN/16000 a=fmtp:111 bitrate=16000 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:116 AAL2-G726-32/8000 a=rtpmap:115 x-msrta/8000 a=fmtp:115 bitrate=11800 a=rtpmap:97 RED/8000 a=rtpmap:13 CN/8000 a=rtpmap:118 CN/16000 a=rtpmap:101 telephone-event/8000 a=fmtp:101 0-16 a=ptime:20 ACK (OUTGOING) Answering on the last OK 200, we are ready and have established the call based on the Early-Media possibility. (In the SIP ACK you can’t directly see which candidate pairs were chosen from the local site) ACK sip:[email protected];opaque=user:epid:LLsGlnCjYlGGrdmzIoEFUwAA;gruu SIP/2.0 Via: SIP/2.0/TLS 192.168.0.16:54763 Max-Forwards: 70 From: <sip:[email protected]>;tag=c013887c44;epid=0639570a7f To: <sip:[email protected]>;epid=a606e73c89;tag=01c33dbedc Call-ID: ab5a007ca2124e95a227f1c82f58cff9 CSeq: 1 ACK Route: <sip:sip.domain-a.com:5061;transport=tls;opaque=state:Ci.R86b100;lr;ms-route- sig=fgFbc2J6H1DsQORB0QtvkWygilYrA-0Ccj13Tr9fH6LkbccFlf6mOs6wAA> Route: <sip:lyncpool1.internFQDN.local:5061;transport=tls;ms-fe=internalFQDN.domain- a.internal;opaque=state:F;lr;received=10.35.3.27;ms-received-cid=78E300> Route: <sip:lyncedgepool.internFQDN.local:5061;transport=tls;lr> Route: <sip:ocs.domain-b.com:5061;transport=tls;epid=a606e73c89;lr;ms-key- info=AAEAAcUcKOgG4q8UZBDQAVcU1Pt2m_6ekgNlaaYrCW5qBdhWPxt97pJbLWjJqjzzJFfUkk6MJ2eS yIm77AmUE76vtBbPqEm5zKxc8R9DxeBx2FZfM49Y16vsDldXQl4uDazo3BWXJDggmtpWiOWkYwhmW adsT813jJ9JhAIxcPTzltiNbNZ9b0_iffEi6oI6_HlftQCLqKb-n8tq1gIsAtSYw8-o2vGxrswJ8HeIFn6z0TFE- Sd_F76aV8DYmi-NkRPcwknKnBMwDhNKQ2pl2vZ61B9oFlcy9xFQOdLZujQUyZ27T06xfFqzz3H- bPyd92DSPFmvjfcvGx6WGqQHI4FIkz2pe9FIKXi8ZVVJhS6EE65YtcKxvu5LfCxSN5P4ZePtClNSdlyV2FLyOt WYXS72c1xHNM5AT53D2KJjwzRURJ1v--1CLkVuiFfCPe8GSPVbqTLmdNLlmpqtF05UiEa- 9coZDWDdYx41iGUNxfhykgSt2iFoTyjdShLxTg6BbAS5zuf5BY_VUuJwKjx9b0- ecWhYzTKK_JaQDim4WGPFqRx8PAcx4GZtv6OQMAwYA8A5ZeXLQWa0DL_PFq4BVF5HfR_jfyQciEGjV Gsc5BXT0CdprYmXNjSyYLzEG4WzCm4jpdX_SnInhJhZ1SZ8kv9n91oV7O7MVwGyBN6ALBbUVCUugkj5 Nfn0qzv_UKDM;ms-route- sig=bkNIsAXKnsiRmbGYKjO4lps9o6nw6ySSsyRfgI3kTNTO9ouKskkfZA1AAA> Route: <sip:lync2013.hq.domain-b.com:5061;transport=tls;opaque=state:T;lr> User-Agent: UCCAPI/15.0.4667.1000 OC/15.0.4667.1000 (Microsoft Lync) Proxy-Authorization: TLS-DSK qop="auth", realm="SIP Communications Service", opaque="9731FB93", targetname="internalFQDN.domain-a.internal", crand="4d36ef5d", cnum="1229", response="4024acc021fc947a444f0235aca6b55bfe38fccd" Content-Length: 0 - - - - - - - - - - - - - - - - - - - - - Final Call Establishment (START) - - - - - - - - - - - - - - - - - - - - - - INVITE (OUTGOING) Very important 2nd INVITEwill be send. Here the candidates are chosen. This is because EARLY MEDIA was in place. It occurs after 8 seconds. This Message is the 2nd important message for troubleshooting INVITE sip:[email protected];opaque=user:epid:LLsGlnCjYlGGrdmzIoEFUwAA;gruu SIP/2.0 Via: SIP/2.0/TLS 192.168.0.16:54763 Max-Forwards: 70 From: <sip:[email protected]>;tag=c013887c44;epid=0639570a7f To: <sip:[email protected]>;epid=a606e73c89;tag=01c33dbedc Call-ID: ab5a007ca2124e95a227f1c82f58cff9 CSeq: 3 INVITE Route: <sip:sip.domain-a.com:5061;transport=tls;opaque=state:Ci.R86b100;lr;ms-route- sig=fgFbc2J6H1DsQORB0QtvkWygilYrA-0Ccj13Tr9fH6LkbccFlf6mOs6wAA> Route: <sip:lyncpool1.internFQDN.local:5061;transport=tls;ms-fe=internalFQDN.domain- a.internal;opaque=state:F;lr;received=10.35.3.27;ms-received-cid=78E300> Route: <sip:lyncedgepool.internFQDN.local:5061;transport=tls;lr> Route: <sip:ocs.domain-b.com:5061;transport=tls;epid=a606e73c89;lr;ms-key- info=AAEAAcUcKOgG4q8UZBDQAVcU1Pt2m_6ekgNlaaYrCW5qBdhWPxt97pJbLWjJqjzzJFfUkk6MJ2eS yIm77AmUE76vtBbPqEm5zKxc8R9DxeBx2FZfM49Y16vsDldXQl4uDazo3BWXJDggmtpWiOWkYwhmW adsT813jJ9JhAIxcPTzltiNbNZ9b0_iffEi6oI6_HlftQCLqKb-n8tq1gIsAtSYw8-o2vGxrswJ8HeIFn6z0TFE- Sd_F76aV8DYmi-NkRPcwknKnBMwDhNKQ2pl2vZ61B9oFlcy9xFQOdLZujQUyZ27T06xfFqzz3H- bPyd92DSPFmvjfcvGx6WGqQHI4FIkz2pe9FIKXi8ZVVJhS6EE65YtcKxvu5LfCxSN5P4ZePtClNSdlyV2FLyOt WYXS72c1xHNM5AT53D2KJjwzRURJ1v--1CLkVuiFfCPe8GSPVbqTLmdNLlmpqtF05UiEa- 9coZDWDdYx41iGUNxfhykgSt2iFoTyjdShLxTg6BbAS5zuf5BY_VUuJwKjx9b0- ecWhYzTKK_JaQDim4WGPFqRx8PAcx4GZtv6OQMAwYA8A5ZeXLQWa0DL_PFq4BVF5HfR_jfyQciEGjV Gsc5BXT0CdprYmXNjSyYLzEG4WzCm4jpdX_SnInhJhZ1SZ8kv9n91oV7O7MVwGyBN6ALBbUVCUugkj5 Nfn0qzv_UKDM;ms-route- sig=bkNIsAXKnsiRmbGYKjO4lps9o6nw6ySSsyRfgI3kTNTO9ouKskkfZA1AAA> Route: <sip:lync2013.hq.domain-b.com:5061;transport=tls;opaque=state:T;lr> Contact: <sip:[email protected];opaque=user:epid:2b02BQuhtlW-2_O2vbNPYAAA;gruu> User-Agent: UCCAPI/15.0.4667.1000 OC/15.0.4667.1000 (Microsoft Lync) Supported: ms-dialog-route-set-update Supported: timer Supported: histinfo Supported: ms-safe-transfer Supported: ms-sender Supported: ms-early-media Supported: 100rel Ms-Conversation-ID: AdAQcNCXWtgHtgCCTziAoXiNeIlHgg== ms-keep-alive: UAC;hop-hop=yes ms-subnet: 192.168.0.0 ms-endpoint-location-data: NetworkScope;ms-media-location-type=Internet Proxy-Authorization: TLS-DSK qop="auth", realm="SIP Communications Service", opaque="9731FB93", targetname="internalFQDN.domain-a.internal", crand="d51c2cac", cnum="1236", response="ebe65f76db398bc77cbcd0023f8a08ebc5383fc3" Content-Type: application/sdp Content-Length: 1238 v=0 o=- 0 2 IN IP4 178.26.121.167 s=session c=IN IP4 178.26.121.167 b=CT:99980 t=0 0 a=x-devicecaps:audio:send,recv;video:send,recv (Chosen CODEC for this Call) m=audio 10668 RTP/SAVP 114 9 112 111 0 8 116 115 97 13 118 101 a=x-ssrc-range:862104576-862104576 a=rtcp-fb:* x-message app send:dsh recv:dsh a=rtcp-rsize a=label:main-audio a=x-source:main-audio a=ice-ufrag:ccwh a=ice-pwd:8X3nfjTthYjuA8vhf+z2FYqM (Chosen candidate pair from the CALLER, the local client initiated the call) a=candidate:9 1 UDP 1862270719 178.26.121.167 10668 typ prflx raddr 192.168.0.16 rport 10668 a=candidate:9 2 UDP 1862270462 178.26.121.167 10669 typ prflx raddr 192.168.0.16 rport 10669 a=x-candidate-info:9 network-type=wlan a=crypto:2 AES_CM_128_HMAC_SHA1_80 inline:eanuVGjZ1PGHP9Eh17H9u5uAr36WW2pjXzrH0s7W|2^31|1:1 (Chosen candidate pair from the CALLEE, the target client received the call) a=remote-candidates:1 188.111.10.69 56186 2 188.111.10.69 52054 a=maxptime:200 a=rtpmap:114 x-msrta/16000 a=fmtp:114 bitrate=29000 a=rtpmap:9 G722/8000 a=rtpmap:112 G7221/16000 a=fmtp:112 bitrate=24000 a=rtpmap:111 SIREN/16000 a=fmtp:111 bitrate=16000 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:116 AAL2-G726-32/8000 a=rtpmap:115 x-msrta/8000 a=fmtp:115 bitrate=11800 a=rtpmap:97 RED/8000 a=rtpmap:13 CN/8000 a=rtpmap:118 CN/16000 a=rtpmap:101 telephone-event/8000 a=fmtp:101 0-16 a=ptime:20 TRYING (INCOMING) Repeated, where targeted client is provide the processing answer to the Caller. SIP/2.0 100 Trying ms-user-logon-data: RemoteUser Authentication-Info: TLS-DSK qop="auth", opaque="9731FB93", srand="2017C6C8", snum="1250", rspauth="29bd3cee53c7ee7d5d2d16447e10c322889787a9", targetname="internalFQDN.domain- a.internal", realm="SIP Communications Service", version=4 From: <sip:[email protected]>;tag=c013887c44;epid=0639570a7f To: <sip:[email protected]>;epid=a606e73c89;tag=01c33dbedc Call-ID: ab5a007ca2124e95a227f1c82f58cff9 CSeq: 3 INVITE Via: SIP/2.0/TLS 192.168.0.16:54763;received=178.26.121.167;ms-received-port=54763;ms-received- cid=86B100 Server: http%3A%2F%2Fwww.microsoft.com%2FLCS%2FUdcAgent Content-Length: 0 OK (INCOMING) The system reports the OPPOSITE (Callee) Sites candidates. This Message is the 3rd important message for troubleshooting SIP/2.0 200 OK ms-user-logon-data: RemoteUser Authentication-Info: TLS-DSK qop="auth", opaque="9731FB93", srand="66092504", snum="1251", rspauth="50704114ce2af2b0b2d8e6baba7beea7385ffde7", targetname="internalFQDN.domain- a.internal", realm="SIP Communications Service", version=4 Via: SIP/2.0/TLS 192.168.0.16:54763;received=178.26.121.167;ms-received-port=54763;ms-received- cid=86B100 Content-Length: 1198 From: <sip:[email protected]>;tag=c013887c44;epid=0639570a7f To: <sip:[email protected]>;epid=a606e73c89;tag=01c33dbedc Call-ID: ab5a007ca2124e95a227f1c82f58cff9 CSeq: 3 INVITE Record-Route: <sip:lync2013.hq.domain-b.com:5061;transport=tls;opaque=state:T;lr> Record-Route: <sip:ocs.domain-b.com:5061;transport=tls;epid=a606e73c89;lr;ms-key- info=AAEAAcUcKOgG4q8UZBDQAVcU1Pt2m_6ekgNlaaYrCW5qBdhWPxt97pJbLWjJqjzzJFfUkk6MJ2eS yIm77AmUE76vtBbPqEm5zKxc8R9DxeBx2FZfM49Y16vsDldXQl4uDazo3BWXJDggmtpWiOWkYwhmW adsT813jJ9JhAIxcPTzltiNbNZ9b0_iffEi6oI6_HlftQCLqKb-n8tq1gIsAtSYw8-o2vGxrswJ8HeIFn6z0TFE- Sd_F76aV8DYmi-NkRPcwknKnBMwDhNKQ2pl2vZ61B9oFlcy9xFQOdLZujQUyZ27T06xfFqzz3H- bPyd92DSPFmvjfcvGx6WGqQHI4FIkz2pe9FIKXi8ZVVJhS6EE65YtcKxvu5LfCxSN5P4ZePtClNSdlyV2FLyOt WYXS72c1xHNM5AT53D2KJjwzRURJ1v--1CLkVuiFfCPe8GSPVbqTLmdNLlmpqtF05UiEa- 9coZDWDdYx41iGUNxfhykgSt2iFoTyjdShLxTg6BbAS5zuf5BY_VUuJwKjx9b0- ecWhYzTKK_JaQDim4WGPFqRx8PAcx4GZtv6OQMAwYA8A5ZeXLQWa0DL_PFq4BVF5HfR_jfyQciEGjV Gsc5BXT0CdprYmXNjSyYLzEG4WzCm4jpdX_SnInhJhZ1SZ8kv9n91oV7O7MVwGyBN6ALBbUVCUugkj5 Nfn0qzv_UKDM;ms-route-sig=bkvaDEc_X- NU45W3umMb4t6ECCG3ZPk6GzArzXrrEbNiM1W1p7kfZA1AAA> Record-Route: <sip:lyncedgepool.internFQDN.local:5061;transport=tls;lr> Record-Route: <sip:lyncpool1.internFQDN.local:5061;transport=tls;ms-fe=internalFQDN.domain- a.internal;opaque=state:F;lr;received=10.35.3.27;ms-received-cid=78E300> Record-Route: <sip:sip.domain-a.com:5061;transport=tls;opaque=state:Ci.R86b100;lr;ms-route- sig=fgOWxUm0EfQarzH0G-ErSJjxVC7XTryjyILvQLZVI1lzTVGglL6mOs6wAA> Contact: <sip:[email protected];opaque=user:epid:LLsGlnCjYlGGrdmzIoEFUwAA;gruu> User-Agent: UCCAPI/15.0.4667.1000 OC/15.0.4667.1000 (Microsoft Lync) Supported: histinfo Supported: ms-safe-transfer Supported: ms-dialog-route-set-update ms-endpoint-location-data: NetworkScope;ms-media-location-type=Intranet Content-Type: application/sdp ms-edge-proxy-message-trust: ms-source-type=DirectPartner;ms-ep- fqdn=lyncedgepool.internFQDN.local;ms-source-network=federation;ms-source-verified- user=verified v=0 o=- 0 2 IN IP4 188.111.10.69 s=session c=IN IP4 188.111.10.69 b=CT:99980 t=0 0 a=x-devicecaps:audio:send,recv;video:send,recv (Target acknowledgement from chosen CODEC for this Call) m=audio 56186 RTP/SAVP 114 9 112 111 0 8 116 115 97 13 118 101 a=x-ssrc-range:76626436-76626436 a=rtcp-fb:* x-message app send:dsh recv:dsh a=rtcp-rsize a=label:main-audio a=x-source:main-audio a=ice-ufrag:MNHU a=ice-pwd:5kvPXMuiOnrb7F2p4DG15DQm (Target site acknowledgement from chosen candidates for this Call) a=candidate:4 1 UDP 184547839 188.111.10.69 56186 typ relay raddr 10.6.0.5 rport 29624 a=candidate:4 2 UDP 184547326 188.111.10.69 52054 typ relay raddr 10.6.0.5 rport 29625 a=crypto:2 AES_CM_128_HMAC_SHA1_80 inline:qoMfVcC0DhPKn3Fs7/9n2BL2iM0P4XSgpxeXR874|2^31|1:1 (Initiator seen IP address @HomeOffice remote acknowledgement from chosen candidates for this Call) a=remote-candidates:1 178.26.121.167 10668 2 178.26.121.167 10669 a=maxptime:200 a=rtcp:52054 a=rtpmap:114 x-msrta/16000 a=fmtp:114 bitrate=29000 a=rtpmap:9 G722/8000 a=rtpmap:112 G7221/16000 a=fmtp:112 bitrate=24000 a=rtpmap:111 SIREN/16000 a=fmtp:111 bitrate=16000 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:116 AAL2-G726-32/8000 a=rtpmap:115 x-msrta/8000 a=fmtp:115 bitrate=11800 a=rtpmap:97 RED/8000 a=rtpmap:13 CN/8000 a=rtpmap:118 CN/16000 a=rtpmap:101 telephone-event/8000 a=fmtp:101 0-16 a=ptime:20 ACK (OUTGOING) Initiator acknowledge the call setup! ACK sip:[email protected];opaque=user:epid:LLsGlnCjYlGGrdmzIoEFUwAA;gruu SIP/2.0 Via: SIP/2.0/TLS 192.168.0.16:54763 Max-Forwards: 70 From: <sip:[email protected]>;tag=c013887c44;epid=0639570a7f To: <sip:[email protected]>;epid=a606e73c89;tag=01c33dbedc Call-ID: ab5a007ca2124e95a227f1c82f58cff9 CSeq: 3 ACK Route: <sip:sip.domain-a.com:5061;transport=tls;opaque=state:Ci.R86b100;lr;ms-route- sig=fgFbc2J6H1DsQORB0QtvkWygilYrA-0Ccj13Tr9fH6LkbccFlf6mOs6wAA> Route: <sip:lyncpool1.internFQDN.local:5061;transport=tls;ms-fe=internalFQDN.domain- a.internal;opaque=state:F;lr;received=10.35.3.27;ms-received-cid=78E300> Route: <sip:lyncedgepool.internFQDN.local:5061;transport=tls;lr> Route: <sip:ocs.domain-b.com:5061;transport=tls;epid=a606e73c89;lr;ms-key- info=AAEAAcUcKOgG4q8UZBDQAVcU1Pt2m_6ekgNlaaYrCW5qBdhWPxt97pJbLWjJqjzzJFfUkk6MJ2eS yIm77AmUE76vtBbPqEm5zKxc8R9DxeBx2FZfM49Y16vsDldXQl4uDazo3BWXJDggmtpWiOWkYwhmW adsT813jJ9JhAIxcPTzltiNbNZ9b0_iffEi6oI6_HlftQCLqKb-n8tq1gIsAtSYw8-o2vGxrswJ8HeIFn6z0TFE- Sd_F76aV8DYmi-NkRPcwknKnBMwDhNKQ2pl2vZ61B9oFlcy9xFQOdLZujQUyZ27T06xfFqzz3H- bPyd92DSPFmvjfcvGx6WGqQHI4FIkz2pe9FIKXi8ZVVJhS6EE65YtcKxvu5LfCxSN5P4ZePtClNSdlyV2FLyOt WYXS72c1xHNM5AT53D2KJjwzRURJ1v--1CLkVuiFfCPe8GSPVbqTLmdNLlmpqtF05UiEa- 9coZDWDdYx41iGUNxfhykgSt2iFoTyjdShLxTg6BbAS5zuf5BY_VUuJwKjx9b0- ecWhYzTKK_JaQDim4WGPFqRx8PAcx4GZtv6OQMAwYA8A5ZeXLQWa0DL_PFq4BVF5HfR_jfyQciEGjV Gsc5BXT0CdprYmXNjSyYLzEG4WzCm4jpdX_SnInhJhZ1SZ8kv9n91oV7O7MVwGyBN6ALBbUVCUugkj5 Nfn0qzv_UKDM;ms-route- sig=bkNIsAXKnsiRmbGYKjO4lps9o6nw6ySSsyRfgI3kTNTO9ouKskkfZA1AAA> Route: <sip:lync2013.hq.domain-b.com:5061;transport=tls;opaque=state:T;lr> User-Agent: UCCAPI/15.0.4667.1000 OC/15.0.4667.1000 (Microsoft Lync) Proxy-Authorization: TLS-DSK qop="auth", realm="SIP Communications Service", opaque="9731FB93", targetname="internalFQDN.domain-a.internal", crand="d8f974c2", cnum="1237", response="6418c1105e802e73602c177cc5c600bff28dde55" Content-Length: 0 UPDATE (OUTGOING) Additional update information are send to the target User B. UPDATE sip:[email protected];opaque=user:epid:LLsGlnCjYlGGrdmzIoEFUwAA;gruu SIP/2.0 Via: SIP/2.0/TLS 192.168.0.16:54763 Max-Forwards: 70 From: <sip:[email protected]>;tag=c013887c44;epid=0639570a7f To: <sip:[email protected]>;epid=a606e73c89;tag=01c33dbedc Call-ID: ab5a007ca2124e95a227f1c82f58cff9 CSeq: 4 UPDATE Route: <sip:sip.domain-a.com:5061;transport=tls;opaque=state:Ci.R86b100;lr;ms-route- sig=fgFbc2J6H1DsQORB0QtvkWygilYrA-0Ccj13Tr9fH6LkbccFlf6mOs6wAA> Route: <sip:lyncpool1.internFQDN.local:5061;transport=tls;ms-fe=internalFQDN.domain- a.internal;opaque=state:F;lr;received=10.35.3.27;ms-received-cid=78E300> Route: <sip:lyncedgepool.internFQDN.local:5061;transport=tls;lr> Route: <sip:ocs.domain-b.com:5061;transport=tls;epid=a606e73c89;lr;ms-key- info=AAEAAcUcKOgG4q8UZBDQAVcU1Pt2m_6ekgNlaaYrCW5qBdhWPxt97pJbLWjJqjzzJFfUkk6MJ2eS yIm77AmUE76vtBbPqEm5zKxc8R9DxeBx2FZfM49Y16vsDldXQl4uDazo3BWXJDggmtpWiOWkYwhmW adsT813jJ9JhAIxcPTzltiNbNZ9b0_iffEi6oI6_HlftQCLqKb-n8tq1gIsAtSYw8-o2vGxrswJ8HeIFn6z0TFE- Sd_F76aV8DYmi-NkRPcwknKnBMwDhNKQ2pl2vZ61B9oFlcy9xFQOdLZujQUyZ27T06xfFqzz3H- bPyd92DSPFmvjfcvGx6WGqQHI4FIkz2pe9FIKXi8ZVVJhS6EE65YtcKxvu5LfCxSN5P4ZePtClNSdlyV2FLyOt WYXS72c1xHNM5AT53D2KJjwzRURJ1v--1CLkVuiFfCPe8GSPVbqTLmdNLlmpqtF05UiEa- 9coZDWDdYx41iGUNxfhykgSt2iFoTyjdShLxTg6BbAS5zuf5BY_VUuJwKjx9b0- ecWhYzTKK_JaQDim4WGPFqRx8PAcx4GZtv6OQMAwYA8A5ZeXLQWa0DL_PFq4BVF5HfR_jfyQciEGjV Gsc5BXT0CdprYmXNjSyYLzEG4WzCm4jpdX_SnInhJhZ1SZ8kv9n91oV7O7MVwGyBN6ALBbUVCUugkj5 Nfn0qzv_UKDM;ms-route- sig=bkNIsAXKnsiRmbGYKjO4lps9o6nw6ySSsyRfgI3kTNTO9ouKskkfZA1AAA> Route: <sip:lync2013.hq.domain-b.com:5061;transport=tls;opaque=state:T;lr> Contact: <sip:[email protected];opaque=user:epid:2b02BQuhtlW-2_O2vbNPYAAA;gruu> User-Agent: UCCAPI/15.0.4667.1000 OC/15.0.4667.1000 (Microsoft Lync) Supported: ms-dialog-route-set-update Supported: timer Session-Expires: 720;refresher=uac Min-SE: 720 Proxy-Authorization: TLS-DSK qop="auth", realm="SIP Communications Service", opaque="9731FB93", targetname="internalFQDN.domain-a.internal", crand="dd88107f", cnum="1330", response="f6cf51543bc9b1fd2a1536957698e3477505fa83" Content-Length: 0 OK (INCOMING) The ACK on the update commend. SIP/2.0 200 OK ms-user-logon-data: RemoteUser Authentication-Info: TLS-DSK qop="auth", opaque="9731FB93", srand="13DC564A", snum="1344", rspauth="8d9dafdf5f39125c29ad76741d2d705863bee0f1", targetname="internalFQDN.domain- a.internal", realm="SIP Communications Service", version=4 Via: SIP/2.0/TLS 192.168.0.16:54763;received=178.26.121.167;ms-received-port=54763;ms-received- cid=86B100 From: <sip:[email protected]>;tag=c013887c44;epid=0639570a7f To: <sip:[email protected]>;epid=a606e73c89;tag=01c33dbedc Call-ID: ab5a007ca2124e95a227f1c82f58cff9 CSeq: 4 UPDATE Record-Route: <sip:lync2013.hq.domain-b.com:5061;transport=tls;opaque=state:T;lr> Record-Route: <sip:ocs.domain-b.com:5061;transport=tls;epid=a606e73c89;lr;ms-key- info=AAEAAcUcKOgG4q8UZBDQAVcU1Pt2m_6ekgNlaaYrCW5qBdhWPxt97pJbLWjJqjzzJFfUkk6MJ2eS yIm77AmUE76vtBbPqEm5zKxc8R9DxeBx2FZfM49Y16vsDldXQl4uDazo3BWXJDggmtpWiOWkYwhmW adsT813jJ9JhAIxcPTzltiNbNZ9b0_iffEi6oI6_HlftQCLqKb-n8tq1gIsAtSYw8-o2vGxrswJ8HeIFn6z0TFE- Sd_F76aV8DYmi-NkRPcwknKnBMwDhNKQ2pl2vZ61B9oFlcy9xFQOdLZujQUyZ27T06xfFqzz3H- bPyd92DSPFmvjfcvGx6WGqQHI4FIkz2pe9FIKXi8ZVVJhS6EE65YtcKxvu5LfCxSN5P4ZePtClNSdlyV2FLyOt WYXS72c1xHNM5AT53D2KJjwzRURJ1v--1CLkVuiFfCPe8GSPVbqTLmdNLlmpqtF05UiEa- 9coZDWDdYx41iGUNxfhykgSt2iFoTyjdShLxTg6BbAS5zuf5BY_VUuJwKjx9b0- ecWhYzTKK_JaQDim4WGPFqRx8PAcx4GZtv6OQMAwYA8A5ZeXLQWa0DL_PFq4BVF5HfR_jfyQciEGjV Gsc5BXT0CdprYmXNjSyYLzEG4WzCm4jpdX_SnInhJhZ1SZ8kv9n91oV7O7MVwGyBN6ALBbUVCUugkj5 Nfn0qzv_UKDM;ms-route-sig=bk4GduDCNxTNV3mUGDqcJjehpo3xWD5UmIeBGvMW- NeXDEkP2SkfZA1AAA> Record-Route: <sip:lyncedgepool.internFQDN.local:5061;transport=tls;lr> Record-Route: <sip:lyncpool1.internFQDN.local:5061;transport=tls;ms-fe=internalFQDN.domain- a.internal;opaque=state:F;lr;received=10.35.3.27;ms-received-cid=78E300> Record-Route: <sip:sip.domain-a.com:5061;transport=tls;opaque=state:Ci.R86b100;lr;ms-route- sig=fgj43gLcqUIWe1otRyx4hGc_E9OAMd7xHVAsuYSVhQcIaGEQYJ6mOs6wAA> Contact: <sip:[email protected];opaque=user:epid:LLsGlnCjYlGGrdmzIoEFUwAA;gruu> User-Agent: UCCAPI/15.0.4667.1000 OC/15.0.4667.1000 (Microsoft Lync) Content-Length: 0 ms-edge-proxy-message-trust: ms-source-type=DirectPartner;ms-ep- fqdn=lyncedgepool.internFQDN.local;ms-source-network=federation;ms-source-verified- user=verified -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-. Call will run until one party will hang up (terminate) the call BYE (INCOMING) The Callee has terminated the Call. We use the ms-client-diagnostic header again identifying the cause for this BYE command. BYE sip:178.26.121.167:54763;transport=tls;ms-opaque=db7d07b19e;ms-received-cid=86B100;grid SIP/2.0 ms-user-logon-data: RemoteUser Via: SIP/2.0/TLS 10.90.0.2:5061;branch=z9hG4bKB8D8E46B.F59FA977F07432BC;branched=FALSE;ms-internal- info="baQuVQtczC98qPtJrOqR8R1AJE4HQqffr8hhp_Y7fO4Ql3qZ_124V57wAA" Via: SIP/2.0/TLS 10.35.3.27:52179;branch=z9hG4bK1F95CEEE.3CB853283194E2BF;branched=FALSE;ms-received- port=52179;ms-received-cid=869C00 Via: SIP/2.0/TLS 10.35.3.30:55234;branch=z9hG4bK4992B5B9.BCE538E3F07342BC;branched=FALSE;ms-received- port=55234;ms-received-cid=144D400 Via: SIP/2.0/TLS 10.20.5.123:50226;branch=z9hG4bK0A74D9E0.7924E7EBB84F72C7;branched=FALSE;ms-internal- info="bgffnqo7oiCfaV0RCJNRTid6CcdhFzn1sjL6rFUa4opADr5yR5HWfZVAAA";received=188.111.10.67 ;ms-received-port=50226;ms-received-cid=8A1900 Via: SIP/2.0/TLS 10.1.1.73:57073;branch=z9hG4bK219343E9.4B78C2AF25BE42BF;branched=FALSE;ms-received- port=57073;ms-received-cid=2DEC800 Via: SIP/2.0/TLS 10.6.0.5:49501;ms-received-port=49501;ms-received-cid=D71300 Max-Forwards: 65 Authentication-Info: TLS-DSK qop="auth", opaque="9731FB93", srand="C01DE96B", snum="1354", rspauth="8d5b523e9aa16aa6905468f4ad3f9268a89463eb", targetname="internalFQDN.domain- a.internal", realm="SIP Communications Service", version=4 Content-Length: 0 From: "" <sip:[email protected]>;epid=a606e73c89;tag=01c33dbedc To: <sip:[email protected]>;tag=c013887c44;epid=0639570a7f Call-ID: ab5a007ca2124e95a227f1c82f58cff9 CSeq: 1 BYE User-Agent: UCCAPI/15.0.4667.1000 OC/15.0.4667.1000 (Microsoft Lync) ms-client-diagnostics: 51004; reason="Action initiated by user";UserType="Callee";MediaType="audio";ICEWarn="0x20";LocalSite="10.6.0.5:9450";LocalMR=" 188.111.10.69:56186";RemoteSite="178.26.121.167:11603";RemoteMR="195.145.140.92:54427";Po rtRange="1025:65000";LocalMRTCPPort="58236";RemoteMRTCPPort="54427";LocalLocation="2";Re moteLocation="1";FederationType="1";NetworkName="hq.domain- b";Interfaces="0x2";BaseInterface="0x2";BaseAddress="10.6.0.5:29624";Protocol="0";LocalInterface ="0x2";LocalAddrType="2";LocalAddress="188.111.10.69:56186";RemoteAddress="178.26.121.167:1 0668";RemoteAddrType="5";MrDnsU="lyncedge2013.hq.domain-b.com";MrResU="0" ms-edge-proxy-message-trust: ms-source-type=DirectPartner;ms-ep- fqdn=lyncedgepool.internFQDN.local;ms-source-network=federation;ms-source-verified- user=verified OK (OUTGOING) The Call termination was done correctly and is acknowledge from the User A with a simple 200 OK SIP/2.0 200 OK Via: SIP/2.0/TLS 10.90.0.2:5061;branch=z9hG4bKB8D8E46B.F59FA977F07432BC;branched=FALSE;ms-internal- info="baQuVQtczC98qPtJrOqR8R1AJE4HQqffr8hhp_Y7fO4Ql3qZ_124V57wAA" Via: SIP/2.0/TLS 10.35.3.27:52179;branch=z9hG4bK1F95CEEE.3CB853283194E2BF;branched=FALSE;ms-received- port=52179;ms-received-cid=869C00 Via: SIP/2.0/TLS 10.35.3.30:55234;branch=z9hG4bK4992B5B9.BCE538E3F07342BC;branched=FALSE;ms-received- port=55234;ms-received-cid=144D400 Via: SIP/2.0/TLS 10.20.5.123:50226;branch=z9hG4bK0A74D9E0.7924E7EBB84F72C7;branched=FALSE;ms-internal- info="bgffnqo7oiCfaV0RCJNRTid6CcdhFzn1sjL6rFUa4opADr5yR5HWfZVAAA";received=188.111.10.67 ;ms-received-port=50226;ms-received-cid=8A1900 Via: SIP/2.0/TLS 10.1.1.73:57073;branch=z9hG4bK219343E9.4B78C2AF25BE42BF;branched=FALSE;ms-received- port=57073;ms-received-cid=2DEC800 Via: SIP/2.0/TLS 10.6.0.5:49501;ms-received-port=49501;ms-received-cid=D71300 From: <sip:[email protected]>;epid=a606e73c89;tag=01c33dbedc To: <sip:[email protected]>;tag=c013887c44;epid=0639570a7f Call-ID: ab5a007ca2124e95a227f1c82f58cff9 CSeq: 1 BYE User-Agent: UCCAPI/15.0.4667.1000 OC/15.0.4667.1000 (Microsoft Lync) Proxy-Authorization: TLS-DSK qop="auth", realm="SIP Communications Service", opaque="9731FB93", targetname="internalFQDN.domain-a.internal", crand="d8c758cf", cnum="1340", response="a34f135db9c88f7f1a2fa38756e6a68886923ff4" Content-Length: 0 Troubleshooting IM, Calls with A/V This is the chapter where we will discuss several common issues and provide information and solutions. We are starting with the most common issue where clients cannot establish an audio/ video connection and learn about the MS-DIAGNOSTICS and MS-CLIENT-DIAGNOSTICS. The last section in this chapter we will talk about the Monitoring reports, especially about identifying bad call quality issues. First, we need having a look into the Address Exchange, the way how the communication path can be established. AV Address Exchange, negotiation of candidates If it is would be possible that client could always communicate straight, meaning in fully routed, flat network, there would be no need for identifying possible communication paths. In the real world are several scenarios where client could be located.  Internal LAN <-> Internal LAN  Internal LAN <-> External (Internet), behind NAT  External (with/ without NAT) <-> External (with/ without NAT)  Federation Scenarios As we understood, NAT could hinder a possible communication, therefore a technique is required exchanging data for client location. This is when the client starts a communication and identify all possible IP addresses and ports. This process is based on Interactive Connectivity Establishment (ICE). This is the process behind the scene, evaluation the most possible direct path. ICE is based on two protocols, Session Traversal Utilities for NAT (STUN) and Traversal Using Relay NAT (TURN). The next illustration demonstrates the exchange, negotiation and acknowledgement of candidates. Making it more understandable, I have linked the process to the related SIP messages in the aforementioned chapter: SIP INVITE -INVITE the USER (OUTGOING) 183 SESSION PROGRESS - PROGRESS REPORT (INCOMING) 200 OK - OK (INCOMING) the candidate exchange is marked in “blue” How this candidates are identified, we have learned from the first illustration. Having a view now into a scenario, where two client are outside the local network. If the both clients are remote, but within the same LAN, they could communicate directly (BLUE). Since the illustration shows the client are in different remote locations, the next possible path would a communication path directly between their Remote Location NAT devices (DARK BLUE). If we assume for any reason that this path wouldn’t be accessible too, the both remote clients need to go via their Edge Server. In the Edge Server scenario I have to point to the dynamic IP port range (PINK) vs. the TCP 443 and UDP 3478 behavior (RED and GREEN). The dynamic ports having a higher priority compare to the fixed TCP/UDP port and at the same I highlight the priority of UDP before TCP. The scenario where the client cannot communicate over the dynamic port range will not fall back to the fixed UDP/ TCP port. Explaining the fall back to the fix port range. You have to understand, that the Edge Server will establish a client connection via the fixed ports, but internally (here the Edge Server itself) will continue using the dynamic port range (ORANGE), meaning within its software component. If you have a pool of Edge server, this communication will once again reach the dynamic port range and communicate with the other Edge server via the dynamic ports on their external NIC. Therefore the DMZ environment in a pool setup requires the external NICs to be able to communicate with each other on the dynamic range. h1 h1 UDP 3478 h1 TCP 443 UDP/ TCP h1 50.000 h1 h2 h2 h2 h2 UDP/ TCP h2 59.999 Let’s explain what is happening here: As we can see from the excerpt above, the ICE negotiation starts with the INVITE and IP Port candidates are submitted to the second client. Next the recipient will reply with it’s own IP Port candidates and pack those into the 182 SESSION PROGRESS message. Thereafter which the two clients will attempt connectivity checks to determine which of those candidates can be used. This process is based on logic within the clients. Once the checks have been passed, the call will be acknowledged with a 200 OK. This message contains the final pair of candidates which are now used. This I had described before. In troubleshooting, you are now able to test those IPs and Ports. If you will not see the 200 OK message, this indicated the check was done, but the connectivity could not be established and your must see instead a BYE message. Most likely this is due to a firewall problem, where those chosen ports are blocked, or the allowed firewall direction is not correctly. The UDP port 3478 must be bi-directional. Another quick look into the final candidates and I will explain SRFLX and PRFLX. FINAL CANDIDATES in the SECOND INVITE a=candidate:9 1 UDP 1862270719 178.26.121.167 10668 typ prflx raddr 192.168.0.16 rport 10668 a=candidate:9 2 UDP 1862270462 178.26.121.167 10669 typ prflx raddr 192.168.0.16 rport 10669 a default a Candidate list b c MRAS d b e c Media d Relay e local remote Local Client NAT Device Edge Server Reflexive addresses are discovered be connecting to the edge server and submitted to the client server reflexive address (srflx) - translated address on the public side of the NAT (obtained from either a STUN server or a TURN server, the Lync or Skype for Business Edge Server). A candidate whose IP address and port are a binding allocated by a NAT for an agent when it sent a packet through the NAT to a server. Server reflexive candidates can be learned by STUN servers using the Binding request, or TURN servers, which provides both a relayed and server reflexive candidate. peer reflexive address (prflx) - A candidate whose IP address and port are a binding allocated by a NAT for an agent when it sent a STUN Binding request through the NAT to its peer. RFC - Interactive Connectivity Establishment (ICE): A Protocol for Network Address Translator (NAT) Traversal for Offer/Answer Protocols https://tools.ietf.org/html/rfc5245 Audio Video Call failed with ms-client-diagnostics (one client is external): One of the most common issue I have seen during the last couple of year is this following error: ms-client-diagnostics: 23; reason="Call failed to establish due to a media connectivity failure when one endpoint is internal and the other is remote";CalleeMediaDebug="audio:ICEWarn=0x80012b REMEMBER: This is a call problem where one client is internal and the other client is a REMOTE client. It results in a call could not be established, even in some constellations IM is working fine, but not audio/ video is possible. Or you simply can’t join a conference call. It will not be stablished. In Lync or Skype for Business client you receive an error warning that a call could not establish due to network problems. Please consult your System Administrator. It might be a bit tricky sporting this issue, because it not be immediately identified where, or better on which site this issue is. WARNING/ ADVICE: If this issue is happen between two client, where you are federated with, you only can approach your own site for troubleshooting. Most likely you don’t have access to your partner’s site. Therefore being advised, test your local site first, than approach your partner and work with him together. Troubleshooting approach:  First get a logging at least from your local client (the client having the issue)  Next, identify the call, is this conference call or a peer-to-peer call This is helping you seeing if the issue is Edge+Client, only Edge or only Client related  Identify the location of the involved client, e.g. internal LAN, Internet, NATed Internet (Home Office)  Start drawing the infrastructure Start the Snooper and get your log file ready as described in the chapter: Client Tracing Log-File location: Identify the call affected and filter based on the CallID. You have now all relevant Session commands consolidated and you are able finding the communication path possibilities, called a=candidates. Candidates always come in pairs. They represent the possible UDP and TCP and TCP-ACT endpoint connection IPs. Your find the following IP’s:  local client IP’s  your NAT device external IP  the Edge Server external IP  with all IPs the remote ports (rport), the local host (host), relay remote address (relay raddr) and server reflexive remote address (srflx raddr) Local host UDP IPv4 address: a=candidate:1 1 UDP 2130706431 192.168.0.16 10668 typ host a=candidate:1 2 UDP 2130705918 192.168.0.16 10669 typ host Local host UDP IPv6 address: a=x-candidate-ipv6:2 1 UDP 2130705919 fd00:26:5bcb:fb92:d812:2961:203d:afd3 15206 typ host a=x-candidate-ipv6:2 2 UDP 2130705406 fd00:26:5bcb:fb92:d812:2961:203d:afd3 15207 typ host a=x-candidate-ipv6:3 1 UDP 33553407 2001:0:5ef5:79fd:34ae:19fd:4de5:8658 23084 typ host a=x-candidate-ipv6:3 2 UDP 33552894 2001:0:5ef5:79fd:34ae:19fd:4de5:8658 23085 typ host Local host TCP-PASS IPv4 address: Edge AV ext IP=195.145.140.92 – local NAT device ext IP=178.26.121.167 a=candidate:4 1 TCP-PASS 174455295 195.145.140.92 54427 typ relay raddr 178.26.121.167 rport 11603 a=candidate:4 2 TCP-PASS 174454782 195.145.140.92 54427 typ relay raddr 178.26.121.167 rport 11603 Local host UDP IPv4 address (relay raddr): Edge AV ext IP=195.145.140.92 – local NAT device ext IP=178.26.121.167 a=candidate:5 1 UDP 184547327 195.145.140.92 57962 typ relay raddr 178.26.121.167 rport 6098 a=candidate:5 2 UDP 184546814 195.145.140.92 51825 typ relay raddr 178.26.121.167 rport 6099 Local host UDP IPv4 address (srflx raddr): NAT device ext IP=178.26.121.167 – local client IP=192.168.0.16 a=candidate:6 1 UDP 1694234111 178.26.121.167 6098 typ srflx raddr 192.168.0.16 rport 6098 a=candidate:6 2 UDP 1694233598 178.26.121.167 6099 typ srflx raddr 192.168.0.16 rport 6099 Local host TCP-ACT IPv4 address (relay raddr): Edge AV ext IP=195.145.140.92 – local NAT device ext IP=178.26.121.167 a=candidate:7 1 TCP-ACT 174846975 195.145.140.92 54427 typ relay raddr 178.26.121.167 rport 11603 a=candidate:7 2 TCP-ACT 174846462 195.145.140.92 54427 typ relay raddr 178.26.121.167 rport 11603 Local host TCP-ACT IPv4 address (srflx raddr): NAT device ext IP=178.26.121.167 – local client IP=192.168.0.16 a=candidate:8 1 TCP-ACT 1684795903 178.26.121.167 11603 typ srflx raddr 192.168.0.16 rport 11603 a=candidate:8 2 TCP-ACT 1684795390 178.26.121.167 11603 typ srflx raddr 192.168.0.16 rport 11603 After the communication path possibilities are identified, you need to find the valid second INVITE where the final candidate pair are exchanged. If this is not the case (the 2nd INVITE does not exist), the issue must be the server and not the network/ client network: FINAL CANDIDATES in the SECOND INVITE a=candidate:9 1 UDP 1862270719 178.26.121.167 10668 typ prflx raddr 192.168.0.16 rport 10668 a=candidate:9 2 UDP 1862270462 178.26.121.167 10669 typ prflx raddr 192.168.0.16 rport 10669 Audio Video Call failed with ms-client-diagnostics: (both client are external): Another common issue is in scenarios where you are federating with a partner and both client are e.g. in their home office. This is a little more complex compared to the scenario described at last. First I have identified the local client and associated Edge server. The Local NAT IP is 178.26.121.167, the local LAN is 192.168.0.16 and the Edge Server 195.145.140.92, all marked in “yellow” and the partner site is marked in “green” with the Edge IP of 212.144.4.102 and the local NAT address 62.75.183.56. ms-client-diagnostics: 27; reason="A federated call failed to establish due to a media connectivity failure where both endpoints are remote"; UserType="Callee";MediaType="audio";ICEWarn="0x8000000";LocalSite="178.26.121.167:5942";Loc alMR="195.145.140.92:54164";RemoteSite="212.144.4.102:50037";RemoteMR="62.75.183.56:5430 7";PortRange="1025:65000";LocalMRTCPPort="54315";RemoteMRTCPPort="54307";LocalLocation=" 1";RemoteLocation="1";FederationType="0";NetworkName="MYNEWHOME";Interfaces="0x14";Bas eInterface="0x4";BaseAddress="192.168.0.16:5942";Protocol="0";LocalInterface="0x4";LocalAddrTy pe="1";LocalAddress="178.26.121.167:5942";RemoteAddress="212.144.4.102:50036";RemoteAddrT ype="1";MrDnsU="sip.partner.nz";MrResU="0" Troubleshooting approach: This must be a peer-to-peer call, with two clients are external, this message can only appear in peer- 2-peer call, since a conference call will always be hosted on MCU, and therefore only a single client will connect to the MCU. Since this is peer to peer, the support it not possible, if not both site are supporting this issue.  First get the logging from both clients, your local and the partners  Next, identify the call, by using the CallID  Identify the location of the involved client, e.g. internal LAN, Internet, NATed Internet (Home Office)  Now find the exchange candidates If this is not helping continue  Import both log’s into SNOOPER Snooper will start drawing the call in a call flow diagram  Validate where the call got interrupted, INVITE, 183 SESSION PROGRESS and 200 OK If you still can’t find any reason why this call was not setup. You need to try and test if in each setup, yours and the partner setup an internal and remote client can talk to each other. Assumingly, you find one setup which is working and one it might not. Going on now, if both sites can make this local vs. remote call there are several other issue. Most likely with the dynamic port range. It could be happen one site blocked the dynamic ports and have a Edge Pool. Than here the Pool site cannot communication from one Edge to the other of the dynamic port range within the DMZ. Else you have a UPNP NAT device at home which is not allocating the “client” communication port proper. Which of UPNP on the remote (home office) router. At last, one of the site have performance issue or run the Edge server virtualized, but did not care about the virtualization guidelines. When all of the troubleshooting approaches fail, you need your NETMON and trace the entire session at least on one site (client) and see where the client try to establish the call over. Maybe TCP or UDP and which target IP address is used. Next step than is, you need to start a NETNOM on one of the Edge Server and trace the network communication from there. Assuming, you had check the firewall and all is absolutely correct. At the end, you will find a port is blocked on one of both edge/ firewall setups. NOTE: I really urge you not using a single IP address on your Edge Server, this makes the troubleshooting more complicated. The aforementioned troubleshooting are based on a setup with a single IP address. You troubleshooting approach need to identify the SIP (ACCESS) and the AV (MRAS) service and with a single IP it makes this very difficult to identify. Diagnostic headers Continuing with the diagnostic headers introduced in the last chapter. These headers can be defined from the SIP Registrar (Servers) and from the SIP clients. They help you identifying issue related to your setup/ configuration. MS-DIAGNOSTICS (Link to ms-diagnostics-header) The following examples I collected, are real-world examples. 1008;reason="Unable to resolve DNS SRV record";domain="gtr-connect-a.com";dns-srv- result="NegativeResult";dns-source="WireQuery";source="sip.domain-a.com" The DNS domain gtr-connect-a.com can’t be resolve. Possible they are not able to federate. The _sipfederationtls._tcp SRV record is not present. 1027;reason="Cannot route this type of SIP request to or from federated partners";source="sip.domain-a.com" Most likely an issue transmitting a SIP command to federated or non-federated partner. Either the IP route is not available, or the target server is busy. 1034;reason="Previous hop federated peer did not report diagnostic information";Domain="inncom.de.de";PeerServer="sip.inncom.de.de";source="sip.domain-a.com" A very common status, where we didn’t received any further information. It also refers to normal status message without the character of an error. 2044;reason="Publication version out of date";source="internalFQDN.domain-a.internal" A SIP session, e.g. IM was much to long open and need to be reestablished. Not an error, just an information that this session hast to be renewed. 2165;reason="Contact subscription is not allowed as the user's contact list has migrated to Exchange.";source="internalFQDN.domain-a.internal" Lync or Skype for Business is not providing the Users BUDDY list, the list is migrated to the Exchange Unified Contact Store and need to be pulled from there. 12006;reason="Trying next hop";source=“Server03.Contoso.com";PhoneUsage="Default Usage";PhoneRoute="External Calling";Gateway="10.111.121.64";appName="OutboundRouting" A status message, informing about the call is not processed, neither timed-out, we need to wait for a proper response. ms-diagnostics: 24100;Component="RTCC/4.0.0.0_ATS/1.0.100";Reason="General diagnostic information.";CalleeICEWarningFlags="Audio:ICEWarn=0x400000,LocalSite=143.111.4.11:39991,Loc alMR=143.111.4.188:50701,RemoteSite=143.111.4.99:39469,RemoteMR=143.111.4.188:58201,Por tRange=49152:57500,LocalMRTCPPort=50701,RemoteMRTCPPort=58201,LocalLocation=2,RemoteL ocation=1,FederationType=0";Source="dcpwplync01.Contoso.com" Just an information about a connection being established on an EDGE server ms-diagnostics: 7037;source="internalFQDN.domain-a.internal";reason="Media stack diagnostics info";component="Audio Video Conferencing Server";CalleeMediaDebugaudio="audio:ICEWarn=0x0,LocalSite=10.35.3.27:49724,LocalMR=195.145 .140.92:51931,RemoteSite=10.35.2.117:13743,RemoteMR=195.145.140.92:59432,PortRange=49152: 57500,LocalMRTCPPort=54292,RemoteMRTCPPort=59432,LocalLocation=2,RemoteLocation=2,Feder ationType=0,Interfaces=0x2,BaseInterface=0x2,BaseAddress=10.35.3.27:51140" An A/V Conferencing statement, that the client has join the conference. In this case not an error, but a source of information used for troubleshooting. ms-diagnostics: 21009;source="xsrvlync5.internFQDN.local";reason="Media stack diagnostics info";component="ASMCU";CalleeMediaDebug="applicationsharing:ICEWarn=0x0,LocalSite=10.35.3. 130:57203,LocalMR=195.145.140.92:55024,PortRange=49152:65535,LocalMRTCPPort=55024,LocalL ocation=0,RemoteLocation=0,FederationType=0,Interfaces=0x2,BaseInterface=0x2,BaseAddress=10. 35.3.130:51614" Audio Server MCU, was contacted for a media stream. MS-CLIENT-DIAGNOSTICS I prefer very often the client site as I have mentioned. This is because all important information are provided in the client log. ms-client-diagnostics: 52094; reason="Instant Messaging conversation terminated on user inactivity" A user terminate / close the IM windows and therefore stopped the communication. ms-client-diagnostics: 51004; reason="Action initiated by user";UserType="Callee";MediaType="audio";ICEWarn="0x0";LocalSite="178.26.121.167:22736";Loc alMR="195.145.140.92:52142";RemoteSite="188.192.77.89:29219";RemoteMR="195.145.140.92:51 797";PortRange="1025:65000";LocalMRTCPPort="55561";RemoteMRTCPPort="51797";LocalLocation ="1";RemoteLocation="1";FederationType="0";NetworkName="MYNEWHOME";Interfaces="0x14";B aseInterface="0x4";BaseAddress="192.168.0.16:22736";Protocol="0";LocalInterface="0x4";LocalAddr Type="1";LocalAddress="178.26.121.167:22736";RemoteAddress="188.192.77.89:25882";RemoteAd drType="1";MrDnsU="sip.Xioppo.nz";MrResU="0" A user initiated a call, where the candidate had been chosen ms-client-diagnostics: 51007;reason="Callee media connectivity diagnosis info";CalleeMediaDebug="application- sharing:ICEWarn=0x0,LocalSite=132.245.0.9:40725,LocalMR=132.245.0.45:52041,RemoteSite=10.35. 2.117:11203,RemoteMR=195.145.140.92:56135,PortRange=50040:50059,LocalMRTCPPort=52041,Re moteMRTCPPort=56135,LocalLocation=1,RemoteLocation=2,FederationType=1,NetworkName=WLA N-E9BE46,Interfaces=0x14,BaseInterface=0x4,BaseAddress=192.168.2.108:50045" The TCP connection information about an application sharing session with the dynamic port ms-client-diagnostics: 51012; reason="Caller timeout on no response";UserType="Callee";MediaType="application- sharing";ICEWarn="0x0";LocalSite="10.35.2.117:25836";LocalMR="195.145.140.92:58980";PortRang e="1025:65000";LocalMRTCPPort="58980";LocalLocation="2";RemoteLocation="0";FederationType= "0";NetworkName="internFQDN.local";Interfaces="0x2";BaseInterface="0x2";BaseAddress="10.35.2. 117:32420;MrDnsU="lyncedgepool.internFQDN.local";MrResU="0";LyncAppSharingDebug="SharerC hannel:0x0; Memory Usage: totalUsedVirtual=1065, availableVirtual=134216662;StartupTime: 2014- 12-03T08:00:35.749Z; An app-sharing call was initiated, but timed out. This could be a network issue or service problem in this case since it is a public IP, on the EDGE server. Or the user had simply not answered the sharing session. Ms-client-diagnostics: 52039; reason="The recipient is responding in another way, such as IM or phone." A A/V call was not answered with voice, instead the user decided to answer with IM or forwarde the call to a voice mail. Ms-client-diagnostics: 52085;reason="Dialog does not exist" A dialog time-out where the session must be reinitiated by the user. Reason: SIP ;cause=488 ;text="Not Acceptable Here" This is a temporarily service problem, were a command is not allowed or can’t be executed on the remote site. ms-client-diagnostics: 52046; reason="Sharer has left the conference";UserType="Callee";MediaType="application- sharing";ICEWarn="0x0";LocalSite="10.35.2.117:28283";LocalMR="195.145.140.92:51876";RemoteSi te="10.35.3.130:50581";RemoteMR="195.145.140.92:55024";PortRange="1025:65000";LocalMRTCP Port="51876";RemoteMRTCPPort="55024";LocalLocation="2";RemoteLocation="2";FederationType= "0";NetworkName="internFQDN.local";Interfaces="0x2";BaseInterface="0x2";BaseAddress="10.35.2. 117:11725";Protocol="1";LocalInterface="0x2";LocalAddrType="0";LocalAddress="10.35.2.117:2663" ;RemoteAddress="10.35.3.130:50581";RemoteAddrType="0";MrDnsU="lyncedgepool.FQDN.local";M rResU="0";LyncAppSharingDebug="ViewerChannel:0x0; Memory Usage: totalUsedVirtual=1030, availableVirtual=134216696; AutoRejoin=0;StartupTime: 2014-12-03T08:00:35.749Z;" Information the user dropped out of the conference, either he left or he had network issue and the TCP session was closed. Monitoring Reports and Call Quality Issues Another efficient method analyzing call events is Monitoring. Lync and Skype for Business provide you multiple reports you can utilize for analyzing Call Quality and other related information’s, like summaries or failure reports. For troubleshooting, we are interested in the Call Quality. First you need to know, the involved client sends an entire report the monitoring services after the session has ended. Those data can be used for further troubleshooting. If a client cannot connect to the monitoring service, the data will be kept until access is possible. Now we have a look into the submitted data from a client. Regardless if a peer-to-peer call or PSTN, as well as conferencing took place, those data is submitted. I will give detailed explanation regarding each relevant data. Next and upfront some explanation about several voice related parameter: JITTER: Jitter (ms) measures the variability of packet delay and results in a distorted or choppy audio experience. packet loss rate: Packet Loss (%) represents the % of packets that did not make it to their destination. Packet loss will cause the audio to be distorted or missing (on the receiver end). network MOS degradation: network MOS degradation is an integer represents the amount of the MOS value lost to network affects. concealed samples ratio: Concealing audio samples is a technique used to deal with dropped network packets. The following table describes measurements and thresholds for bad call analyzes and identification. Table 1. Events, descriptions, and measurements/thresholds (take from NEXTHOP/ HelpFile) Caused By Event Description Measurements - Thresholds Network Network Send Packet loss and jitter on Jitter: Good <20ms, Bad >30ms Quality receive stream is severe Packet Loss: Good <3%, Bad >7% and introducing distortion Network Receive Concealed packet ratio Concealed Packet Ratio: Good Quality on send stream is severe <2%, Bad >3% and introducing distortion Low Bandwidth Available bandwidth is Dynamic based on codec insufficient for acceptable voice/video experience High Latency Network latency is RTT: Good <300ms, Bad >500ms severe and preventing interactive communication Machine Low CPU cycles Insufficient CPU for Flag when audio processing current encoding/decoding engine is not modalities and getting sufficient CPU cycles applications, causing audio distortion Remote user Low SNR Poor capture quality Flag if participant in the from remote user; conference has a noisy distortion from noise or environment user being too far from microphone Echo Remote user's device or Flag if remote user (or participant setup is causing echo in a conference) has a device beyond the ability of the setup that is injecting echo into system to compensate the call Device Echo Device or setup is * Timestamp noise causing echo beyond the * Dynamic and Adaptive NLP ability of the system to attenuation compensate * Post-AEC echo percentage * Microphone clipping due to far- end signal Howling Audio feedback loop Check for howling/screeching detected (caused by from other endpoints in the room multiple endpoints sharing audio path) Capture Device Microphone currently Check capture buffer status Not Functioning used is not functioning  correctly, causing one- way audio issues Render Device Speaker currently used Check render buffer status Not is not functioning correctly, causing one- Functioning way audio issues Render Glitches Severe glitches in audio Look for glitches after adaptive rendering, causing render buffer distortion; can be caused by driver issues, deferred procedure call (DPC) storm (drivers), high CPU usage Low SNR Poor capture quality; Low SNR distortion from noise or High absolute noise level after user being too far from AGC microphone Microphone User's speech level is Microphone clipping during near Clipping too high for the system end-only portions to handle and is causing distortion Near End to Echo User's speech is too low Near end-to-echo ratio Ratio compared to the echo being captured, limits Speaker volume to high or too far ability to interrupt a from the microphone user Half Duplex To prevent echo, system Flag the event when device is in Mode enters half duplex mode "voice switch" mode (dynamic switching between render and capture streams), which limits ability to interrupt a user Multiple Audio Multiple audio Detect conference join tone in Endpoints endpoints detected in the room the same session, system compensates by reducing render volume The meaning of AVARAGE: Most of the parameters are measured with average values. This values cannot provide you with information about specific periods of time within a call. For example, if a user reported bad quality during a call, saying with last for about 20min. The bad user experience was within a period of 2-3 min, while the rest of the call was acceptable. The average will not provide you the data you might need to report to the user the bad network connectivity he experienced in the midst of the call. Example: Submitting Metrics after Conference call The reporting message is identical will “normal” SIP message. 12/15/2014|08:57:21.131 6DC:C20 INFO :: Sending Packet - 195.145.140.92:5061 (From Local Address: 192.168.0.16:61813) 6069 bytes: 12/15/2014|08:57:21.131 6DC:C20 INFO :: SERVICE sip:lyncpool1.domain-a.local@domain- a.com;gruu;opaque=srvr:HomeServer:PMmip8HdslKOQd6hXyAMwgAA SIP/2.0 Via: SIP/2.0/TLS 192.168.0.16:61813 Max-Forwards: 70 From: <sip:[email protected]>;tag=ac2bb40467;epid=0639570a7f To: <sip:[email protected];gruu;opaque=srvr:HomeServer:PMmip8HdslKOQd6hXyAMwgAA> Call-ID: 884fcce5fcaf422a950d081204b034a8 CSeq: 1 SERVICE Contact: <sip:[email protected];opaque=user:epid:2b02BQuhtlW-2_O2vbNPYAAA;gruu> User-Agent: UCCAPI/15.0.4675.1000 OC/15.0.4675.1000 (Microsoft Lync) Proxy-Authorization: TLS-DSK qop="auth", realm="SIP Communications Service", opaque="6B133F97", targetname="xsrvlync7.domain-a.local", crand="7e8a7e88", cnum="354", response="36a80167fec98270630fd6ffda03814004d562d0" Content-Type: application/vq-rtcpxr+xml (this indicates the reporting message) Content-Length: 5242 - <VQReportEvent xmlns="ms-rtcp-metrics" v2="ms-rtcp-metrics.v2" SchemaVersion="2.0"> - <VQSessionReport SessionId="5be5fbf45d97434eb594018bcc142400;from-tag=663e541128;to-tag=ee30759290"> (this indicates the reporting client data) <Endpoint xmlns="ms-rtcp-metrics" v2="ms-rtcp-metrics.v2" v3="ms-rtcp-metrics.v3" Name="xclient-Poett" OS="Windows 6.2.9200 SP: 0.0 Type: 1(Workstation) Suite: 0000000000000100 Arch: x64 WOW64: False" CPUName="Intel(R) Core(TM) i7-4510U CPU @ 2.00GHz" CPUNumberOfCores="2" CPUProcessorSpeed="2594" VirtualizationFlag="0"/> (this indicates CallID, and can be used for tracking a call e.g. Snooper) - <DialogInfo CallId="5be5fbf45d97434eb594018bcc142400" FromTag="663e541128" ToTag="ee30759290" Start="2014-12-15T07:56:52.0455Z" End="2014-12-15T07:57:21.0009Z"> <FromURI>sip:[email protected]</FromURI> (UC user) <ToURI>sip:[email protected];gruu;opaque=app:conf:applicationsharing:id:PTRL3DQ4</ToURI> (this indicates the calling target, in this case the conference call) <Caller>true</Caller> <LocalContactURI>sip:[email protected];opaque=user:epid:2b02BQuhtlW- 2_O2vbNPYAAA;gruu</LocalContactURI> <RemoteContactURI>sip:thomas.poett@domain- a.com;gruu;opaque=app:conf:applicationsharing:id:PTRL3DQ4</RemoteContactURI> <LocalUserAgent>UCCAPI/15.0.4675.1000 OC/15.0.4675.1000 (Microsoft Lync)</LocalUserAgent> (this informs us about the client which was used) <RemoteUserAgent>RTCC/5.0.0.0 applicationsharing</RemoteUserAgent> (since this was a conference call, we see the remote party, in this case the Server Conferencing Application) <ConfURI>sip:Thomas.Poett@domain- a.com;gruu;opaque=app:conf:applicationsharing:id:PTRL3DQ4</ConfURI> <MediationServerBypassFlag>false</MediationServerBypassFlag> <Separator/> <RegisteredInside>false</RegisteredInside> - </DialogInfo> - <MediaLine xmlns="ms-rtcp-metrics" v2="ms-rtcp-metrics.v2" v3="ms-rtcp-metrics.v3" Label="data"> - <Description> - <Connectivity> (this is the most important section, it indicates connection points and network information) <Ice>DIRECT</Ice> <IceWarningFlags>32</IceWarningFlags>  - <RelayAddress> <IPAddr>195.145.140.92</IPAddr> (Edge Server AV IP Address for ICE) <Port>50126</Port> (Edge Server AV IP PORT Address for ICE) - </RelayAddress> - </Connectivity> <Security>SRTP</Security> <Transport>TCP</Transport> - <NetworkConnectivityInfo> <NetworkConnection>wifi</NetworkConnection> <VPN>false</VPN> <LinkSpeed>144000000</LinkSpeed> <BSSID>10-BF-48-4A-F6-BE</BSSID> <Separator/> <NetworkConnectionDetails>Wifi</NetworkConnectionDetails> <WifiDriverDeviceDesc>Intel(R) Dual Band Wireless-AC 7260;Microsoft Wi-Fi Direct Virtual Adapter</WifiDriverDeviceDesc> <WifiDriverVersion>Intel:16.5.3.6;Microsoft:6.3.9600.16384</WifiDriverVersion> - </NetworkConnectivityInfo> - <LocalAddr> <IPAddr>192.168.0.16</IPAddr> (local client IP Address) <Port>20723</Port> (local client IP Port Address) <SubnetMask>255.255.255.0</SubnetMask> <MACAddr>A0-A8-CD-8A-BC-22</MACAddr> - </LocalAddr> - <RemoteAddr> <IPAddr>195.145.140.92</IPAddr> (Edge AV IP Address) <Port>51512</Port> (Edge AV IP PORT Address) - </RemoteAddr> - <ReflexiveLocalIPAddress> <IPAddr>178.26.121.167</IPAddr> (local client external IP Address [behind NAT]) <Port>4144</Port>(local client external IP Port Address) - </ReflexiveLocalIPAddress> <MidCallReport>false</MidCallReport> <Separator3/> - </Description> (This section provides information about the incoming data stream) - <InboundStream Id="2086245205"> - <Network> - <Jitter> (Jitter data) <InterArrival>0</InterArrival> <InterArrivalMax>1</InterArrivalMax> <InterArrivalSD>0</InterArrivalSD> - </Jitter> - <Utilization> <Packets>10</Packets> - </Utilization> <Separator/> - </Network> - <Payload> <ApplicationSharing/> - </Payload> - </InboundStream> (This section provides information about the outgoing data stream) As we will see in this data, the used feature was application sharing. We can identify the network utilization (bandwidth), frame rate, as well as the session contained a “shared desktop”) - <OutboundStream Id="2982043061"> - <Network> - <Jitter> <InterArrival>4</InterArrival> <InterArrivalMax>31</InterArrivalMax> <InterArrivalSD>8.246211</InterArrivalSD> - </Jitter> - <Delay> <RoundTrip>28</RoundTrip> <RoundTripMax>42</RoundTripMax> - </Delay> - <Utilization> <Packets>532</Packets> <BandwidthEst>330707</BandwidthEst> <BandwidthEstMin>2261425</BandwidthEstMin> <BandwidthEstMax>2261425</BandwidthEstMax> <BandwidthEstStdDev>0</BandwidthEstStdDev> <BandwidthEstAvge>2261425</BandwidthEstAvge>  - </Utilization> <Separator/> <LossCongestionPercent>0</LossCongestionPercent> <DelayCongestionPercent>0</DelayCongestionPercent> <ContentionDetectedPercent>0</ContentionDetectedPercent> - </Network> - <Payload> - <ApplicationSharing> <ApplicationShared>Desktop</ApplicationShared> <Separator/> <NumSharingStarted v4="ms-rtcp-metrics.v4">1</NumSharingStarted> <NumRemoteControlChanges v4="ms-rtcp-metrics.v4">0</NumRemoteControlChanges> - <SharerAppSharingEstablishTime> <SignalingTime v4="ms-rtcp-metrics.v4">718</SignalingTime> <MediaSetupTime v4="ms-rtcp-metrics.v4">3469</MediaSetupTime> <ProtocolConnectTime v4="ms-rtcp-metrics.v4">641</ProtocolConnectTime> - </SharerAppSharingEstablishTime> - <ScrapingFrameRate> <Average>25.000000</Average> - </ScrapingFrameRate> - <OutgoingTileRate> <Average>56.809975</Average> - <Burst> <Occurrences>0</Occurrences> <Density>0.000000</Density> <Duration>0.000000</Duration> - </Burst> - <Gap> <Occurrences>1</Occurrences> <Density>56.809975</Density> <Duration>23.844000</Duration> - </Gap> - </OutgoingTileRate> - <CaptureTileRate> <Average>223.967117</Average> - <Burst> <Occurrences>0</Occurrences> <Density>0.000000</Density> <Duration>0.000000</Duration> - </Burst> - <Gap> <Occurrences>1</Occurrences> <Density>223.967117</Density> <Duration>24.452999</Duration> - </Gap> - </CaptureTileRate> - </ApplicationSharing> - </Payload> - </OutboundStream> <AppliedBandwidthLimit>1495000</AppliedBandwidthLimit> <AppliedBandwidthSource>ReceiveSideTURN</AppliedBandwidthSource> - </MediaLine> - </VQSessionReport> 12/15/2014|08:57:21.131 6DC:C20 INFO :: End of Sending Packet - 195.145.140.92:5061 (From Local Address: 192.168.0.16:61813) 6069 bytes Software Defined Networking (SDN) Lync and Skype for Business provide the new SDN API 3rd party developers can utilize. Generally is supports the End-2-End monitoring including all involved elements from Software down to the Network. It supports you troubleshooting efforts and also offers an solution to pre-detect upcoming issues. Jamie Stark spoke about the SDN on the last Lync Conference in Las Vegas, where he had this perfect illustration, making the usability for SDN more visible to you. I don’t want to get deeper into the SDN, just offering you the better understanding, why it might support your troubleshooting efforts. Preventing Configuration and other Issues (Testing Commands) Lync and Skype for Business provides several management shell integrated TEST commands. There are a couple of tests you can run. Here I provide some of the most common test you can run. Most of the tests require valid user accounts for testing purposes. Thats why you should have a few test user accounts setup in your AD. e.g. TSTUSR01-10. Assign those users also different policies, identically with the policies assigned to your production users. This let you simulate dedicated scenarios where user could report issues and you can do a direct testing of those related configurations. It an important task that you test your services before you take you environment or configuration into production. Only if you can consider your synthetic test as successful, you can consider in the event of reported issues, that you will advanced with other troubleshooting aspects instead of seeking inside Lync/ Skype for Business. For the aforementioned test user the most test commands required the test user to be authenticated. With the following variable, you can store the users credentials and utilize those within the test commands. $cred1 = Get-Credential "domain-a\TSTUSR01" $cred2 = Get-Credential "domain-a\TSTUSR02" Take Away: You should during your troubleshooting take the test commands into you considerations, because they help providing information’s you would else bother users for or would not be able to receive from users on the spot. IM Starting with the standard features, Instant Message and Presence. You should validate if the Client- to-Server-to-Client communication can work. First you should test the ability for presence. There its an approach is the Server/ Pool can handle Presence state. Test-CsPresence -TragetFqdn Test-CsPresence -TargetFqdn POOLFQDN -SubscriberSipAddress "sip:TSTUSR01@domain- a.com" -SubscriberCredential $cred1 -PublisherSipAddress "sip:TSTUSR02@domain- a.com" -PublisherCredential $cred2 Next is the Instant Message. As explained, the IM communication path is always via the user home pool server. Test-CsIM Test-CsIM -TargetFqdn POOLFQDN -SubscriberSipAddress "sip:[email protected]" - SubscriberCredential $cred1 -PublisherSipAddress "sip:[email protected]" - PublisherCredential $cred2 Voice Next we focus on Voice. Voice itself is involved within the most common communication scenarios. Either as Peer-to-Peer call or within in any kind of Multi-Point conference. Enterprise Voice is generally seen the same data stream, but has the PSTN/ SIP Trunk telephony involved. Therefore I have separate the test command into this sub-categories. Voice (P2P) Confirms that users are able to make peer to peer calls (signal only). Meaning is will not test the Voice datastream, instead it test the SIP Signaling. Test-CsP2PAV -TargetFqdn POOLFQDN -SubscriberSipAddress "sip:[email protected]" -SubscriberCredential $cred1 -PublisherSipAddress "sip:[email protected]" - PublisherCredential $cred2 Voice (Enterprise Voice) Enterprise Voice, the most complex topic in UC implementation. In this chapter I take care about your Lync/ Skype for Business setup only. With those commands 3rd party systems, e.g. Gateways or SIP Trunks cannot be tested. Here you validate your Enterprise Voice concept and the implementation you applied. The best approach to validate and analyze results of your configuration, you are entitled setting up Configuration Tests. Those test can than run periodically and supports your effort running the PBX functionality in Lync/ Skype for Business successfully. I list the required commands managing this test. You should read the help –file setting up those test according to your need. First you manage the configuration tests with the following commands: Get-CsVoiceTestConfiguration New-CsVoiceTestConfiguration Remove-CsVoiceTestConfiguration Set-CsVoiceTestConfiguration Running the synthetic test: Test-CsVoiceTestConfiguration The test command can “standalone” it need to have test cases submitted, therefore you are required reading those cases into a variable (array). This will most apply to all test commands. You can either utilize the pipe (|) or array scripting feature. The following example show you how you could utilize those VoiceTestConfiguration: $a = Get-CsVoiceTestConfiguration -Identity TestConfig1 Test-CsVoiceTestConfiguration -TestCaseInputObject $a $dp = Get-CsDialPlan -Identity TSTUSR01-GERMANY $vp = Get-CsVoicePolicy -Identity TSTUSR-ALL Test-CsVoiceTestConfiguration -DialedNumber 0895645342 -Dialplan $dp - VoicePolicy $vp The next listed tests, will support you gathering information about individual elements configured within your enterprise voice setup, e.g. DialPlans, Normalization Rules or Routes Here you validated if the dialed number can be normalized for users how have this DP assigned. (e.g. here all users in the site MUNICH, which have the automatic dailpan parameter activated) $a = Get-CsDialPlan -Identity site:Munich Test-CsDialPlan -DialedNumber 0895645342 -Dialplan $a | Format-List With in DialPlans you find your defined Normalization Rules. If you want to test individual NR’s outside of a DialPlan, you have to run this commands. The Dialed number will be than tested against the chose NR. $a = Get-CsVoiceNormalizationRule -Identity "global/11 digit number rule" Test-CsVoiceNormalizationRule -DialedNumber 5645342 -NormalizationRule $a The Voice Policy test runs a dialed and normalized number agains a VP and validates if thise number is allowed for a dial out. $a = Get-CsVoicePolicy -Identity MunichLocal Test-CsVoicePolicy -TargetNumber "+49895645342" -VoicePolicy $a Voice Routes decides if a call with has a target assigned can be utilized. $vr = Get-CsVoiceRoute -Identity testroute Test-CsVoiceRoute -TargetNumber "+49895645342" -Route $vr Most important in your Enterprise Voice setup is, if a user will match the expected and designed calling behavior. This command do not required credentials. I recommend for dedicated user groups setting an Excel based test sheet, which you run in dedicated scripts. Now I Test-CsVoiceUser -DialedNumber "+49895645342" -SipUri "sip:[email protected]" -Verbose The Location Information Service can be included in your tests, but are not part of my troubleshooting guide. Not a lot of implementations I have supported had LIS implemented. Other tests which can dig even deeper into the setup and calls can fully initiated I have listed here. Test-CSOutboundCall – Validates policies, signaling and media to the PSTN TestCSPSTNPeertoPeerCall – Establishes call between two Lync users over PSTN Test-CsExumConnectivity (ExumConnectivity) – Confirms users are able to connect to Unified Messaging Conferencing As described during the Voice troubleshooting, in conferencing we have similar challenges. Internal conferencing is mainly subject to configuration / misconfiguration issues, while it makes everything more complex if we involve the external/ remote component via the Edge server. Internally, you can test with several management shell commands, in addition to the next level of troubleshooting by using the OCSLogging tool. Externally, you need to validate two more systems, the Reverse Proxy publishing the MEET URL and the Edge server publishing the Web Conferencing and AV. Authentication should have led to an earlier issue while a remote client would have logged-in. The following commands will support the internal testing’s: Test-CsAVConference (AvConference) – Test users are able to create an AV conference call Test-CsASConference (ASConference) – Test users are able to create an Application Sharing conference call Test-CsDialInConferencing (Phone Dialin Conference) – Test a dialin possibility for a conference Test-CsGroupIM (escalated IM Group chat) – Test a IM conference Next we have a looking OCSLogger. If you need to analyze conferencing, it has several components the so called MCU (Multipoint Connection Unit) involved. Therefore the OCSLogger can log events happened inside those applications. In the following illustration the AsMCU (Application Sharing) and AvMCU (Audio Video) is marked. All other MCU are available for logging too, e.g. AvMP, ImMcu WEB Services Web Service are segregated into internal and external Web Sites, as well for High Availability, they required Load Balancing. This makes it important having an eye onto them too. Services like the Address Book, or Group Expansion are served by IIS. This is also happened with the Mobility Services. The interesting parameter is the –EXTERNAL, where you define running the test against the 4443 associated web site. Test-CsGroupExpansion -TargetFqdn PoolFqdn -GroupEmailAddress DL-LYNC- TST@domain-a-com You can add -UserCredentials testing the correct authentication, including the authentication method. $cred1 = Get-Credential "domain-a\TSTUSR01" Test-CsAddressBookWebQuery -TargetFqdn PoolFqdn -UserCredential $cred1 - UserSipAddress "sip:[email protected]" -TargetSipAddress "sip:[email protected]" EDGE (external/ remote) Edge Server, one system component where the most integrators have problems with. Here I urge you strongly receiving the client log file and analyze the issue from here. Generally, after a deployment you should test the audio functionality before you assign users. During my past years supporting UC environments, the most issues were found on the involved firewalls due to port blockings and misunderstanding of the client direct connection to the Edge server. That’s why test the routings as well. But back to the available test commands. Test-CsAVEdgeConnectivity (AVEdgeConnectivity) – Test edge server is able to accept connections for peer to peer calls $cred = Get-Credential "domain-a\TSTUSR01" Test-CsAVEdgeConnectivity -TargetFqdn PoolFqdn -UserSipAddress "sip:[email protected]" -UserCredential $cred I also recommend writing the results into a log file OutLoggerVariable. Health Monitoring Test User Last but not least, during the aforementioned chapters, the test commands were introduced with user credentials. There is one more option, where in professional environment user are pre- configured. In Lync/ Skype for Business this option is called Health Monitoring. This are collection of monitors including test user with their credentials. A collection will be defined with the following commands: New-CsHealthMonitoringConfiguration -Identity PoolFqdn -FirstTestUserSipUri "sip:[email protected]" -SecondTestUserSipUri "sip:tstusr02@domain- a.com" With this Health Monitors you can simplify Test-Cs command by using a short cut, e.g. Test-CsPresence -TragetFqdn Troubleshooting Exchange Integration Exchange integration covers several topics. We have the UCS, the Unified Contact Store. There is most asked topic, the setup and support for EWS, the Exchange Web Service integration. The other both areas are the IM integration into OWA, Outlook Web Apps and the UM integration (Unified Messaging). All of those interaction between Lync/ Skype for Business and Exchange have different aspects and use different technologies. In my personal blog http://lyncuc.blogspot.com, the EWS blog is the most requested article. So I try to focus on this troubleshooting aspect a little more in detail. Lync and Skype for Business require three very important task to be configured correctly: - Correct integration of Exchange Autodiscover They are directly related to the EWS Service - Certificates making OAuth and MTLS communication possible - Establishing Lync/ Skype for Business and Exchange partner applications Most common issue here can be identified, that Exchange Admin have very little understanding of a correct configuration of Autodiscover. This is based on the fact that even wrongly configured Autodiscover setups are covered by the so called SCP (Service Connection Points) in Active Directory. Outlook clients, as well as Exchange Server can make use of the SCP Entry in AD and do not need to query this service throughout DNS. Therefore, you as support engineer for Lync/ Skype for Business should make clear how important the full understanding of Exchange is. Personally have very seldom seen that UC was the point of failure. Again, certificates are very must essential in secure communication. This also includes the correct certificates for all partner applications like Exchange. In this troubleshooting chapter you should read all for sections, since several topics are not repeated over again. Verify Exchange AutoDiscover setup First validate the internal and external DNS settings: Both the internal as well as the external SIP Domain should be identically and has to be configured with the following entries. autodiscover.domain.name CNAME exchangeserver(CAS) _autodiscover._tcp.domain.name SRV 0 0 443 exchangeserver(CAS) ewsurl.domain.name A exchangeserver (CAS) Remember here, Exchange 2013 do not provide you with the configurable CAS Array parameter. Instead Exchange 2013 utilize DNS load balancing or DNS in conjunction with a Hardware Load Balancer. Here the CAS URL can either be set to multiple Exchange CAS Server or to the HLB VIP address. The AutoDiscover is defined and configured with its own command: Set-AutodiscoverVirtualDirectory -Identity 'autodiscover (default Web site)' -ExternalURL 'https://ews.domain.name/autodiscover/autodiscover.xml' -InternalURL 'https://ews.domain.name/autodiscover/autodiscover.xml' -BasicAuthentication $true The AutodiscoverVirtualDirectoy URL are supposed for Microsoft's optional use only. Therefore it is not necessary and not Best-Practice defining them! If you set the URL's, it will NOT HAVE AN IMPACT, but it supports your troubleshooting process, since you can use them as a “reminder”. The last important verification which has to be check is: Autodiscover and EWS service do NOT support FBA (form based authentication). You might like checking my detailed article: http://lyncuc.blogspot.de/2013/01/lync-and-exchange-web-services-ews-and.html Exchange Unified Contact Store Integration UCS is a central storage integration for Lync/ Skype for Business users buddy list and the Archiving in the same database/ mailbox of a Exchange user assotiated with Lync/ Skype for Business. The first essential part is the bi-directional partner application esablishment. In Exchange a partner application can only be established via scripts. There is not possibility of identifing this more easy. If you actually not sure if a partner application was allredy setup and the script runs again, an additional LyncEnterpise-ApplicationAccount with an increasing number is generated. Once this was done, you need to correct AD users associated with and corret the Exchange RBAC. This illustration show a account which was accedentally created again: In Active Directoy an Partner Application account is established in the default container (“USERS”). This account is also used in the Exchange RBAC. Therefore now Exchange hast the essential configuration and Lync/ Skype for Business is the associated partner application. From here we go on with the opposite site. In Lync/ Skype for Business you can see the importance of a correct setup for the Exchange Autodiscover. The OAuth configuration requires a correctly working Exchange Autodiscovery setup and is configured in Lync/ Skype for Business in the global Oauth Configuration. Set-CsOAuthConfiguration -Identity global -ExchangeAutodiscoverUrl "https://autodiscover.domain-a.com/autodiscover/autodiscover.svc" Now since authentication is possible we have to establish Exchange as a partner application in Lync/ Skype for Business. This is the second time Exchange Autodiscover comes in to the game. As we see makes sure Exchange is correctly configured and DNS is proper established. DNS SMTP/ SIP Domain split setup is required. Routing is another issue. If DNS returns the external Reverse Proxy IP, make sure this can be a valid path for autodiscover. Validate the REALM, the realm is the DNS name of your AD Domain, not the SMTP or SIP Domain. It is used for authentications. Analyzing the Exchange related Application Pool is quite a hassle and needs to be discussed with the Exchange administrators. Not in all circumstances we can use an Application Pool. In Exchange UM for example it might be required to have trusted computers instead. Generally, the pool would be you Exchange CAS servers, or each individual Exchange CAS server. For each Lync/ Skype for Business central Pool/ Standard server it hast to be setup, if they are in different Side IDs. Therefore check the PoolFqdn for Exchange CAS. In Exchange you must have the PoolFqdn used in Lync/ Skype for Business in the SAN entries, else the MTLS connectivity cannot be established correctly and the validation process is made to fail. Now we make use of the Test commands again. It is assumed, you have the correct Exchange policies assigned to the user you are going to test. Test-CsUnifiedContactStore -UserSipAddress "sip:[email protected]" - TargetFqdn "cie-ly01.domain-a.local" The Test shows: 1. User has no Lync Contacts 2. User has contacts, but the contacts where not jet migrated to UCS 3. After a short while Lync did the migration and show success A common issue is with MTLS setup. But a false positive is, if you don’t run the Management Shell in ADMIN MODE! Error: If this is happened, you must run the Lync Management Shell as Admin, else the Console cannot Access the Private Key for TLS communication The UCS also has some restrictions you should be aware of:  48 pixels by 48 pixels, the size used for the Active Directory thumbnailPhoto attribute. If you upload a photo to Exchange 2013 Exchange will automatically create a 48 pixel by 48 pixel version of that photo and update the user's thumbnailPhoto attribute. Note, however, that the reverse is not true: if you manually update the thumbnailPhoto attribute in Active Directory the photo in the user's Exchange 2013 mailbox will not automatically be updated.  96 pixels by 96 pixels, for use in Microsoft Outlook 2013 Web App, Microsoft Outlook 2013, Microsoft Lync Web App, and Lync 2013.  648 pixels by 648 pixels for use in Lync 2013 and Microsoft Lync Web App. Exchange IM integration on Outlook Web Apps The initial setup for UCS is identically with the IM integration into Outlook Web Apps. Nevertheless, Exchange WEB.CONFIG file had to be modified. After you applied an Exchange CU or Service Pack, this file might have been over written. Therefore you need to check the changes you made during your IM integration work. If the sign-in is working, your experience look like the following. The login process can be traced in Exchange, as well as with OCSLogger. Exchange Web Service Integration Exchange Web Services (EWS) provides the functionality to enable client applications to communicate with the Exchange server. Exchange Web Services (EWS) is a cross-platform API that enables applications to access mailbox items such as email messages, meetings, and contacts from on-premises versions of Exchange. EWS applications can access mailbox items locally or remotely by sending a request in a SOAP-based XML message. The Web Service are configured with their own command, validate the setting by using the following command. Verify if the internal and external URL are correct. Get-WebServicesVirtualDirectory Validate if EWS is globally enabled: Get-OrganizationConfig and see if the parameter EwsEnable is $TRUE Test is the EWS is accessible: https://CASFqdn/EWS/Exchange.asmx Validating the correct authentication settings for EWS and Autodiscover: Service EWS Anonymous Windows authentication authentication AutoDiscover Anonymous Windows Basic authentication authentication authentication You can easily verify if EWS is working correctly on the client site by opening the client configuration setting information. If the master piece “AUTODISCOVER” is configured correctly, the EWS should be fine too. You can identify this if the EWS Internal/ External URL is provide (this feature is provided by AutoDiscover). From there the client make a connection to Exchange EWS and provides you with the information of the EWS is accessible by reporting “EWS Status OK”. Additionally you will find the information if this user was UCS enabled or not “Contact List Provider” is set to “UCS” (Unified Contact Store) Exchange Unified Messaging Integration Before is explain the UM service is detail, I need to inform you that the UM Services changed from Exchange 2007/2010 to Exchange 2013/ 2015 (vNext). Therefore I describe only the process for the actual Exchange versions. Exchange segregate the UM services into two areas, the UMCallRouter and the UMService. While the UMCallRouter acts as “proxy”, if it receives a SIP Invite message, if does a lookup for its recipient. Similar as the CAS Server is doing for user (Outlook or OWA). It than know’s the user mailbox location and sent are SIP REDIRECT answer to the sender (Lync Server), which than is able to establish the SIP connection directly with the user mailbox server. This behavior is illustrated in the call flow diagram below. Exchange MBX Lync/ Exchange CAS UMService.exe / Skype for Business UMCallRouter.exe worker process SIP or secure SIP (TCP 5060/ 5061) SIP REDIRECT UMCallRouter.exe SIP or secure SIP (TCP 5062/5063) UMService.exe SIP 302 Moved temporarily UM Worker Process SIP or secure SIP (TCP 506x) RTP or SRTP traffic The following table illustrates the TCP port usage within a UM deployment. Communication type TCP Port Notes SIP to CAS 5060 (unsecured CAS listen for inbound SIP traffic on UMCallRouterService.exe 5061 (secured) these ports, changeable via Set-UMCallRouterSettings SIP to Mailbox 5062 (unsecured Mailbox role listens for inbound SIP UMService.exe 5063 (secured) from CAS on these ports. They are fixed SIP to UM worker process 5065 & 5067 (unsecured All ports are used when the 5066 & 5068 (secured) UMStartupMode is set to DUAL. If it’s set to TCP or TLS, only 5065 and 5066 are used. Those ports can’t be changed Next step is having a look into the UM Services: Call Answering Call answering is the receiving of voice messages on behalf of users whose calls are not answered or are busy. . Outlook Voice Access Outlook Voice Access enables an Enterprise Voice user to access not just voice mail, but also the Exchange inbox, including email, calendar, and contacts from a telephony interface. The subscriber access number is assigned by an Exchange UM administrator. Auto Attendant Auto attendant is an Exchange UM feature that can be used to configure a phone number that outside users can dial to reach company representatives. Two more important troubleshooting task have to be validate. The first is the numbering format: Please ensure you are utilize the E.164 format. If e.g. Lync or PXB is sending other formats, you might be able to cover those scenarios with dialplans, but the user experience is also impacted by simply showing e.g. wrong extension or entire numbers. Other is the Access Number might not be matched. The second important is the relation between the certificates used for UM/ UM Service and the Lync/ Skype for Business Trusted Application Pools/ Computers. As usual, MTLS is required to configured with the correct CN/SN and SAN if those setup is not matching, the UM will also experience disconnection issues. Coming back to the AA and SA numbers As we can see in the Exchange UM Integration Utility, you can setup either AA or SA, but both need to be associated with a E.164 phone number. You need to trace with OCSLogger calls coming in to those numbers. Lync/ Skype for Business vs. Exchange integration (OWA/ IM) Exchange needs to be trusted by Lync/ Skype for Business. Otherwise the communication will fail. As aforementioned, there are two possible setups for Exchange. In the example illustration below, we assume two different setups: The first is setup with Load Balancer and the CAS Array DNS name ExchCAS.contoso.com and the second example is DNS load balanced service, where we have the individual Exchange CAS computers trusted (CAS03 and CAS04). Regardless this setup here represents also an Exchange consolidated installation, where CAS and MBX is installed on the same server. (Else you need to provide the trusted servers for mbx too) Not only UM is depending on the Trusted Application Server, as well the IM integration into OWA. If the SIP server is now communicating with Exchange it validated the certificates presented by Exchange UMCallRouter and UMService. Therefore the communication is in DNS load balances environment or in HLB environment (single leg configuration) always answered by the host itself. Only if a HLB two-armed solution is used, the HLB will answers including the SSL offloading. Now we see if the server is responding with another name the SN, e.g. you have used the CAS Array configuration, than the individual Server Name (SN) must be trusted. That’s why the trusted computer model is important. If you trace the connections with OCSLogger, you will find SIP message rejections where the certificate is not trusted. Troubleshooting conferences As we all know, we can configure Lync Edge Server in several way. 1) Single Edge Server with a SINGLE IP ADDRESS 2) Single Edge Server with MULTIPLE IP ADDRESSES (3x IPs) 3) Multiple Edge Server in a Pool, with MULTIPLE IP ADDRESSES (Zx 3 IPs) Regardless what we are going to configure, there are common / well-known TCP Port necessary making Lync work, which are: Access: Port: 443 and 5061 Conferencing: Port: 443 and (444 with single IP) AV: Port: 443 (I have not listed other ports, e.g. STUN or the dynamic port range. This is not required for the topic discussed here) Now we need having a look into the Lync Web Conferencing Service, publish via the Edge Server. Looking at the incoming IP connection and there is a different. If you really configure Best-Practice and use three (3) public IP addresses, everything is going to be fine. No one should experience any issue. This is due to the connection made to e.g. conf.customer.com and it's common TCP Port 443 as for incoming. Because this ports are always activated on every Firewall or via any Reverse Proxy. But what happened if we are using the single IP address with single FQDN? As you can see in the config example, we must use another TCP Port rather than 443, because with the single IP, 443 is occupied by AV. Per default, Microsoft suggests TCP Port 444. But regardless of this, whatever port we are going to choose, mostly the outgoing Firewalls are not open for any for those other TCP Ports. (Seen from the prospective of a meeting participant). This clearly means, you will experience issues with a lot of your Federation Partners and meeting participants! NOTE: Beware of the negative impact if you decide going for a SINGLE PUBLIC IP ADDRESS. I do NOT recommend this configuration. Persistent Shared Object Model (PSOM) protocol The client communicates with the Web Conferencing service by using the Persistent Shared Object Model (PSOM) protocol. PSOM is a custom protocol that is used for transporting web conferencing content. PSOM is the web conferencing protocol used for exchanging data collaboration content (white board, Pools, Q&A) and control, listed under the section of MEDIA PROTOCOLS, There are 4 Conferencing Services:  IM Conferencing MCU  Application Sharing MCU  A/V Conferencing MCU  Web Conferencing MCU The only MCU utilizing PSOM is the Web Conferencing service. You will find PSOM on the Edge Server too. Reference: Conferencing Flow Where PSOM is used in detail: During a “Join Meeting”, the client establishes a direct connection with the conferencing service.  If the service is an A/V Conferencing Service, the signaling protocol is SIP and the media is transported over RTP/RTCP. If the service is the Application Sharing Conferencing service, the signaling protocol is SIP and the media is transported over RDP encapsulated within RTP  If the service is the Web Conferencing service, both signaling and media are sent using the PSOM protocol. Lync Server also supports sharing RDP wrapped in RTP PSOM side-by-side for a scenario where features such as desktop sharing (RDP), whiteboard, and polling are used simultaneously. - <conference-view state="full"> + <entity-view state="full" entity="sip:[email protected];gruu;opaque=app:conf:focus:id:K5I89BTR"> + <entity-view state="full" entity="sip:[email protected];gruu;opaque=app:conf:applicationsharing:id:K5I89BTR"> + <entity-view state="full" entity="sip:[email protected];gruu;opaque=app:conf:audio-video:id:K5I89BTR"> + <entity-view state="full" entity="sip:[email protected];gruu;opaque=app:conf:chat:id:K5I89BTR"> + <entity-view state="full" entity="sip:[email protected];gruu;opaque=app:conf:data-conf:id:K5I89BTR"> External FQDN with single IP address: If we are going to choose a single IP address, we would have TCP Port overlapping. Therefore the only way avoiding this is assigning another port. Additionally we will also see and are reminded that Lync highly depends on DNS. If we have single IP, we must have use a single, unique FQDN for all services. ACCESS: SIP.CUSTOMER.COM PORT:5061 CONFERENCING: SIP.CUSTOMER.COM PORT:444 AV: SIP.CUSTOMER.COM PORT:443 External FQDN with multiple IP addresses: In comparison, if we are choosing to make use of three individual IP addresses. We also need three different FQDN, one for each service. ACCESS: SIP.CUSTOMER.COM PORT:443 CONFERENCING: CONF.CUSTOMER.COM PORT:443 AV: AV.CUSTOMER.COM PORT:443 If we now compare with the Microsoft provided illustration of the Edge Server related Enterprise Perimeter Network, this TCP Port named here are for INCOMING CONNECTIONS ONLY. Now it becomes clearer what the requirements are if an outside (remote) Lync user needs a connection to the published services. The most common used services are: IM, Audio/Video, Desktop or App Sharing, as well as Presence Queries. Regardless which configuration was chosen, the single IP or triple IP configuration, those services are all addressed via the common port of 443 and “5061”. So we can assume, those service are mostly working independently of the chosen configuration model. Conference INVITE and ACCESS First access to the conferencing modalities is during the Logon process. We had learned and understood during the login how the authentication will work. The conferencing is first initiated during this process, where as a result the A/V Edge conferencing TCP and UPD ports are exchanged. This exchange is initiated during the SIP SERVICE request and submitted during the 2nd 200/OK. The service request is issued against the conferencing factory. SIP REGISTER 1 200 OK 2 Allow-Events: vnd-microsoft-provisioning,vnd- microsoft-roaming-contacts,... SIP SERVICE 3 xmlns="urn:ietf:params:xml:ns:cccp" to="sip:[email protected];gruu;opaque=app: conf:focusfactory" Access <getConferencingCapabilities server- Edge mode="14"/> 200 OK 4 from="sip:[email protected];gruu;opaque=app: conf:focusfactory" Frontend - <getConferencingCapabilities capability- Server version="0"> 200 OK 7 <hostName>avedge.customer.com <udpPort>3478 <tcpPort>443 <username> 77kuzt8ydfrtz4b52leOF <password>Wnjui8udk87ahsz/FG= <duration>480 MTLS Service 5 MRAS 200 OK 6 A/V Edge Outer Edge Inner Local Client Firewall Server Firewall For conferences of all modalities, the initial join process is the same as in a normal Lync session setup. Lync Server introduced simple URLs, simplifying the URL that is used to join conferences. These URLs, when configured for external participants, are published through a reverse proxy. The simple URL associated with the meeting join process is the Meet Simple URL. When a conference is generated or a scheduled conference is sent through email, the meeting join URL is shared. When a user clicks the meeting URL or types it into a web browser, it connects to the reverse proxy over HTTPS. The reverse proxy then proxies the web request to the configured Director or Front End pool. Next we have a look into the process during a “join a meeting” process. During this process, one interesting information from the client is also submitted. We know that a client has a certificate issued from the server. This certificate is submitted again during the INFO message. The illustration below show the progress of joining a meeting. Where we clearly see that the media flow starts after the conferencing permission and setting are submitted to the joining user/ client.  SIP INVITE mscp="http://schemas ./cccpextensions" C3PVersion="1" to="sip:[email protected];gruu;opaque=app: conf:focus:id:QVSHW1P8" 100 TRYING 183 SESSION PROGRESS 200 INVITE DIALOG CREATED Access Edge ACK and INFO Frontend In INFO: <X509-certificate>MIIB4TCCA .. Server 202 ACCEPTED INFO <getConference> <conference-info entity="sip:[email protected];gruu;opaque= app:conf:focus:id:QVSHW1P8" (all permission, infos and URLs are submitted) STUN A/V MEDIA Edge Outer Edge Inner Local Client Firewall Server Firewall Call flow explanation to the illustration above The Audio and Video Conferencing join experience is similar to the Application Sharing Conferencing join in that the call flow process is nearly identical. The user sends an INVITE to the A/V Conference Service URI, and then performs a series of ICE protocol connectivity checks. This establishes a media path and relays media through the Audio/Video Edge service to the Audio/Video Conferencing service that is hosted on the Front End pool or a dedicated A/V Conferencing Server. Because this process is the same as the Application Sharing Join process, this section highlights only the relevant differences. The major difference between this call flow and the Application Sharing call flow is that a user sends multiple sets of candidates both for audio and video. In the introduction of this chapter we have understood where the conferencing data is send to. “If the service is the Web Conferencing service, both signaling and media are sent using the PSOM protocol.” What we still can see is the access of the PSOM port (443 or single IP address e.g. 444). As explained, this port, where the web conferencing is addressed with, is used for web conference and conference controlling, where the joined client sends commands to define the conference progress. If you joined a conference owned outside of your environment (an anonymous meeting), you connect to the Web Conferencing Service on their Edge Server, which than is PSOM. The SIP messages are flowing through the Web Conferencing Edge Service but do not show the TCP PORT. This can only be traced with NETMON on the Edge server or with CLS/ OCSLogger on the conferencing server (Frontend). Why not Single IP on EDGE Port 444 Problem…. Beside the trace, this is also very nice example of how the Edge service is acting as an Application Proxy, you see how the Edge receiver an internal message, will do the processing and then only it will send the message on behalf out to the internet. I traced a problematic single IP configuration from outgoing point of view: (This TRACE runs through the Web Conferencing Server) This is the Edge Server: The customer clicked an MEETING INVITE in Outlook, the Web Browser opened and was issuing the conference back to the Lync Desktop Client - invited user is identified as [email protected] (aka CALLER participant at this meeting) - internal network is 10.10.x.y with an AD FQDN INTERNAL.AD - meeting initiator is [email protected] and meeting ID is V3JZ92CZ (aka ORGANIZER) - external single IP 99.79.91.241 Edge intern NIC incoming from caller -> organizer INVITE the Edge should initiate the outgoing meeting, seen in the message-body. the conferencing service should add an used (caller) to the meeting TL_INFO(TF_PROTOCOL) [0]097C.0834::07/11/2014-11:15:26.143.0000003d (SIPStack,SIPAdminLog::ProtocolRecord::Flush:ProtocolRecord.cpp(265))[2376455152] $$begin_recordTrace-Correlation-Id: 2376455152 Instance-Id: BF9DB Direction: incoming;source="internal edge";destination="external edge" Peer: LYNCFEPOOL01.INTERNAL.AD:51714 Message-Type: request Start-Line: INVITE sip:[email protected];gruu;opaque=app:conf:focus:id:V3JZ92CZ SIP/2.0 From: "Caller, Nils"<sip:[email protected]>;tag=e4776a37ed;epid=f5710ea2b3 To: <sip:[email protected];gruu;opaque=app:conf:focus:id:V3JZ92CZ> Call-ID: 53fa037467934a3aa58afa7da405cffd CSeq: 1 INVITE Contact:< sip:[email protected];opaque=user:epid:6Ng_wBKilFeryhezW1lEuAAA;gruu> Via: SIP/2.0/TLS 10.10.10.127:51714;branch=z9hG4bKDFE93E20.E0C27AFE227343AD;branched=FALSE Via: SIP/2.0/TLS 10.10.45.69:49360;ms-received-port=49360;ms-received-cid=2E9D600 Record-Route:< sip:LYNCFEPOOL01.INTERNAL.AD:5061;transport=tls;ms- fe=LYNCFRCLSERV01.INTERNAL.AD;opaque=state:T;lr>;tag=0CF71FDEF89C166BEDCEB50B598409B1 Max-Forwards: 69 Content-Length: 1018 Content-Type: application/cccp+xml Message-Body: - <request xmlns="urn:ietf:params:xml:ns:cccp" mscp="http://schemas.microsoft.com/rtc/2005/08/cccpextensions" C3PVersion="1" to="sip:[email protected];gruu;opaque=app:conf:focus:id:V3JZ92CZ" from="sip:[email protected]" requestId="344391952"> + <addUser> </request> Next the domain discovery done by the Edge Server and finding the FQDN and IP TL_INFO(TF_CONNECTION) [3]097C.02C0::07/11/2014-11:15:26.174.000001eb (SIPStack,SIPAdminLog::WriteConnectionEvent:SIPAdminLog.cpp(454))[3899431948] $$begin_recordSeverity: information Text: TLS negotiation started Local-IP: 10.11.10.84:61621 Peer-IP: 99.79.91.241:5061 Peer: sip.singleip.com:5061 Connection-ID: 0x49E800 Transport: M-TLS Here the TLS negotiation INFO message is generated. TL_INFO(TF_DIAG) [0]097C.0C74::07/11/2014-11:15:26.252.00000286 (SIPStack,SIPAdminLog::WriteDiagnosticEvent:SIPAdminLog.cpp(802))[1802118479] $$begin_recordSeverity: information Text: Routed a locally generated request SIP-Start-Line: NEGOTIATE sip:127.0.0.1:5061 SIP/2.0 SIP-Call-ID: 38AA2A4D958FC58A1F97 SIP-CSeq: 1 NEGOTIATE Peer: sip.singleip.com:5061 The Edge Server send the negotiate message the meeting org. TL_INFO(TF_PROTOCOL) [0]097C.0C74::07/11/2014-11:15:26.252.00000292 (SIPStack,SIPAdminLog::ProtocolRecord::Flush:ProtocolRecord.cpp(265))[1802118479] $$begin_recordTrace-Correlation-Id: 1802118479 Instance-Id: BF9DC Direction: outgoing;source="local";destination="external edge" Peer: sip.singleip.com:5061 Message-Type: request Start-Line: NEGOTIATE sip:127.0.0.1:5061 SIP/2.0 From: sip:SIP.CORRECT.COM;tag=6AA3DC66E3BF1C9E7EFA44888B1B7E51 To: sip:sip.singleip.com Call-ID: 38AA2A4D958FC58A1F97 CSeq: 1 NEGOTIATE Via: SIP/2.0/TLS 10.11.10.84:61621;branch=z9hG4bKD7CAB5A3.FA2521EF7066539E;branched=FALSE Max-Forwards: 0 Content-Length: 0 Compression: LZ77-64K Supported: NewNegotiate,OCSNative,ECC,IPv6,TlsRecordSplit Server: RTC/5.0 We now receive the SIP 200/OK message based in the INVITE, so the ACCESS Edge at the caller site is working. TL_INFO(TF_PROTOCOL) [0]097C.0C74::07/11/2014-11:15:26.283.000002bf (SIPStack,SIPAdminLog::ProtocolRecord::Flush:ProtocolRecord.cpp(265))[3194725999] $$begin_recordTrace-Correlation-Id: 3194725999 Instance-Id: BF9DD Direction: incoming;source="external edge";destination="internal edge" Peer: sip.singleip.com:5061 Message-Type: response Start-Line: SIP/2.0 200 OK From: sip:SIP.CORRECT.COM;tag=6AA3DC66E3BF1C9E7EFA44888B1B7E51 To: sip:sip.singleip.com;tag=EDEE8C0427072C271B9B823E3B26BC5F Call-ID: 38AA2A4D958FC58A1F97 CSeq: 1 NEGOTIATE Via: SIP/2.0/TLS 10.11.10.84:61621;branch=z9hG4bKD7CAB5A3.FA2521EF7066539E;branched=FALSE;received=80.157.6.163;ms- received-port=61621;ms-received-cid=D5BD000 Content-Length: 0 Compression: LZ77-64K Supported: NewNegotiate,OCSNative,ECC,TlsRecordSplit Server: RTC/4.0 Edge as Application Proxy must process several Information, here connection is established with the organizer site TL_INFO(TF_CONNECTION) [0]097C.0C74::07/11/2014-11:15:26.283.000002da (SIPStack,SIPAdminLog::WriteConnectionEvent:SIPAdminLog.cpp(383))[3899431948] $$begin_recordSeverity: information Text: Connection established Peer-IP: 99.79.91.241:5061 Peer: sip.singleip.com:5061 Transport: M-TLS Data: alertable="no" Now the Edge has processed even more and also agreed the sip.singleip.com domain, its certificate and established TLS connection TL_INFO(TF_CONNECTION) [0]097C.0C74::07/11/2014-11:15:26.283.0000030a (SIPStack,SIPAdminLog::WriteConnectionEvent:SIPAdminLog.cpp(383))[3899431948] $$begin_recordSeverity: information Text: SIP message traffic has established the peer server as a Discovered Domain federated peer Peer-IP: 99.79.91.241:5061 Peer: sip.singleip.com:5061 Transport: M-TLS Edge internal process info for send INVITE from intern site (caller), domain is now in the discovered domain list TL_INFO(TF_DIAG) [0]097C.0C74::07/11/2014-11:15:26.283.00000310 (SIPStack,SIPAdminLog::WriteDiagnosticEvent:SIPAdminLog.cpp(802))[2376455152] $$begin_recordSeverity: information Text: The message has a Discovered Domain SIP-Start-Line: INVITE sip:[email protected];gruu;opaque=app:conf:focus:id:V3JZ92CZ SIP/2.0 SIP-Call-ID: 53fa037467934a3aa58afa7da405cffd SIP-CSeq: 1 INVITE Peer: sip.singleip.com:5061 Data: domain="singleip.com" Edge is now preparing for sending the INVITE to the external organizer TL_INFO(TF_DIAG) [0]097C.0C74::07/11/2014-11:15:26.283.0000036b (SIPStack,SIPAdminLog::WriteDiagnosticEvent:SIPAdminLog.cpp(802))[2376455152] $$begin_recordSeverity: information Text: Routed a request to a Discovered Domain federated peer SIP-Start-Line: INVITE sip:[email protected];gruu;opaque=app:conf:focus:id:V3JZ92CZ SIP/2.0 SIP-Call-ID: 53fa037467934a3aa58afa7da405cffd SIP-CSeq: 1 INVITE Peer: sip.singleip.com:5061 Here it comes: Edge has now proxied the internal caller sending request he would like to join the external meeting. therefore the caller request is send finally to the external site (singleip.com) TL_INFO(TF_PROTOCOL) [0]097C.0C74::07/11/2014-11:15:26.283.00000377 (SIPStack,SIPAdminLog::ProtocolRecord::Flush:ProtocolRecord.cpp(265))[2376455152] $$begin_recordTrace-Correlation-Id: 2376455152 Instance-Id: BF9DB Direction: outgoing;source="internal edge";destination="external edge" Peer: sip.singleip.com:5061 Message-Type: request Start-Line: INVITE sip:[email protected];gruu;opaque=app:conf:focus:id:V3JZ92CZ SIP/2.0 From: "Caller, Nils"<sip:[email protected]>;tag=e4776a37ed;epid=f5710ea2b3 To: <sip:[email protected];gruu;opaque=app:conf:focus:id:V3JZ92CZ> Call-ID: 53fa037467934a3aa58afa7da405cffd CSeq: 1 INVITE Contact:< sip:[email protected];opaque=user:epid:6Ng_wBKilFeryhezW1lEuAAA;gruu> Via: SIP/2.0/TLS 10.11.10.84:61621;branch=z9hG4bK1616E949.64036B07705F839E;branched=FALSE;ms-internal- info="aqgQ48dd2SfNMeRfbruAAZXq8dFFBTtKluOHag-KpPn1wHawNkNq4BswAA" Via: SIP/2.0/TLS 10.10.10.127:51714;branch=z9hG4bKDFE93E20.E0C27AFE227343AD;branched=FALSE;ms-received- port=51714;ms-received-cid=4B8F00 Via: SIP/2.0/TLS 10.10.45.69:49360;ms-received-port=49360;ms-received-cid=2E9D600 Record-Route:< sip:SIP.CORRECT.COM:5061;transport=tls;epid=f5710ea2b3;lr;ms-key- info=AAEAAdJOgwIBMa2t5ZzPASlkLxWClArLg5fYAz5vMU1--3qvyX7XKhdANCiKC-GE07tJz6E3DmxM-Uo1JCVXZwiNF0uZ2ZM- MBkpzf8q70BVHpEeVVJxW4-ptvp1zWHfjfpaL75- G59cC8TTOSXREQP7w4wTVzV730yNT9Ph48zRr2YVibOrM1R1QJThh3fhOMGY6BjkBdw1rGGmlgbssXVOjCAu7Q9vs3VwxSIOqB6A 1VbZNUG8zoAjDaqm_FdS6cziurxnJSAl9at4yVYFUS7LIzHbhMal7Clz5WDPENfDR- 6YkottO4A0_I4ocqv3P6k_txrZumb8uB5Gf0pnwjZuwy2boSzwgo2aVu-OrvBcaL9IIlRA0kMgZs62YXBCUVl_F7KRJ9cSUpgbN- B5pMVtPhU7nlCZluxkqB-db2B149xOw4aQ4Eyso3c7gRntFMq61dfI3kPyPFDgNdpDtNmgWwcvEBXFCK2l8EGSHElRsNSIyE- D1UgGQBieo3bPW41uxGIXJfndV9nAMQlbB6mqR- UEbwNGyCgX_cbdHEdPQbClzoqvQFDZ9D857BWNaTBAYfVtbstvrVLsx5vvjAuFY_zFDtNjwKZtYkKJRnedDYnv0kJbBK7pu3bw3LQ0W ruFFS-shxBWC9mrUSrhFggcQIoolloakvT0bXL4tHdggWb9fsSSUrCMCQm4KSQC;ms-route- sig=dtgD9HmH2Ck2pYUw_OaiCBzENJLtQyjLBgVnOdt26vsAoHawNkjqWm6wAA>;ms- rrsig=dtATEXIj4kuWMVvcXWz8MoMCB3C4BfDk6UfICkkpSjpRMHawNkjqWm6wAA;tag=6AA3DC66E3BF1C9E7EFA44888B1B7E51 Record-Route:< sip:LYNCFEPOOL01.INTERNAL.AD:5061;transport=tls;ms- fe=LYNCFRCLSERV01.INTERNAL.AD;opaque=state:T;lr>;tag=0CF71FDEF89C166BEDCEB50B598409B1 Max-Forwards: 68 Content-Length: 1018 Content-Type: application/cccp+xml Message-Body: - <request xmlns="urn:ietf:params:xml:ns:cccp" mscp="http://schemas.microsoft.com/rtc/2005/08/cccpextensions" C3PVersion="1" to="sip:[email protected];gruu;opaque=app:conf:focus:id:V3JZ92CZ" from="sip:[email protected]" requestId="344391952"> + <addUser> </request> Immediately after the INVITE was send, the SIP 404 Not Found was received. How this can be happened? The Web Conferencing Server is awaiting incoming request on TCP Port 444, This is REQUEST is coming directly from the initiating client. The local PC's Lync Client. The TCP Port 444 is blocked and the opposite Edge Server now send the INFO that a client did not send a request, meaning he did not receive any request matching on Port 444. (You would see this IP package, if you run a WireShark on our Web Traffic) TL_INFO(TF_PROTOCOL) [0]097C.0C74::07/11/2014-11:15:26.299.000003b3 (SIPStack,SIPAdminLog::ProtocolRecord::Flush:ProtocolRecord.cpp(265))[2376455152] $$begin_recordTrace-Correlation-Id: 2376455152 Instance-Id: BF9DE Direction: incoming;source="external edge";destination="internal edge" Peer: sip.singleip.com:5061 Message-Type: response Start-Line: SIP/2.0 404 Not Found From: "Caller, Nils"<sip:[email protected]>;tag=e4776a37ed;epid=f5710ea2b3 To: <sip:[email protected];gruu;opaque=app:conf:focus:id:V3JZ92CZ>;tag=EDEE8C0427072C271B9B823E3B26BC5F Call-ID: 53fa037467934a3aa58afa7da405cffd CSeq: 1 INVITE Via: SIP/2.0/TLS 10.11.10.84:61621;branch=z9hG4bK1616E949.64036B07705F839E;branched=FALSE;ms-internal- info="aqgQ48dd2SfNMeRfbruAAZXq8dFFBTtKluOHag-KpPn1wHawNkNq4BswAA";received=80.157.6.163;ms-received- port=61621;ms-received-cid=D5BD000 Via: SIP/2.0/TLS 10.10.10.127:51714;branch=z9hG4bKDFE93E20.E0C27AFE227343AD;branched=FALSE;ms-received- port=51714;ms-received-cid=4B8F00 Via: SIP/2.0/TLS 10.10.45.69:49360;ms-received-port=49360;ms-received-cid=2E9D600 Content-Length: 0 Two more processing infos regarding the SIP domain. TL_INFO(TF_DIAG) [0]097C.0C74::07/11/2014-11:15:26.299.0000040f (SIPStack,SIPAdminLog::WriteDiagnosticEvent:SIPAdminLog.cpp(802))[2376455152] $$begin_recordSeverity: information Text: The message has a Discovered Domain SIP-Start-Line: SIP/2.0 404 Not Found SIP-Call-ID: 53fa037467934a3aa58afa7da405cffd SIP-CSeq: 1 INVITE Peer: sip.singleip.com:5061 Data: domain="singleip.com" Preparing the SIP 404 message being send to the internal Lync Frontend. TL_INFO(TF_DIAG) [0]097C.0C74::07/11/2014-11:15:26.299.000004c3 (SIPStack,SIPAdminLog::WriteDiagnosticEvent:SIPAdminLog.cpp(802))[2376455152] $$begin_recordSeverity: information Text: Response successfully routed SIP-Start-Line: SIP/2.0 404 Not Found SIP-Call-ID: 53fa037467934a3aa58afa7da405cffd SIP-CSeq: 1 INVITE Peer: LYNCFEPOOL01.INTERNAL.AD:51714 The proxied message is now send to the internal Frontend. TL_INFO(TF_PROTOCOL) [0]097C.0C74::07/11/2014-11:15:26.299.000004cf (SIPStack,SIPAdminLog::ProtocolRecord::Flush:ProtocolRecord.cpp(265))[2376455152] $$begin_recordTrace-Correlation-Id: 2376455152 Instance-Id: BF9DE Direction: outgoing;source="external edge";destination="internal edge" Peer: LYNCFEPOOL01.INTERNAL.AD:51714 Message-Type: response Start-Line: SIP/2.0 404 Not Found From: "Caller, Nils"<sip:[email protected]>;tag=e4776a37ed;epid=f5710ea2b3 To: <sip:[email protected];gruu;opaque=app:conf:focus:id:V3JZ92CZ>;tag=EDEE8C0427072C271B9B823E3B26BC5F Call-ID: 53fa037467934a3aa58afa7da405cffd CSeq: 1 INVITE Via: SIP/2.0/TLS 10.10.10.127:51714;branch=z9hG4bKDFE93E20.E0C27AFE227343AD;branched=FALSE;ms-received- port=51714;ms-received-cid=4B8F00 Via: SIP/2.0/TLS 10.10.45.69:49360;ms-received-port=49360;ms-received-cid=2E9D600 Content-Length: 0 ms-diagnostics: 1034;reason="Previous hop federated peer did not report diagnostic information";Domain="singleip.com";PeerServer="sip.singleip.com";source="SIP.CORRECT.COM" ms-edge-proxy-message-trust: ms-source-type=AutoFederation;ms-ep-fqdn=EDGEPOOL01.INTERNAL.AD;ms-source-verified- user=unverified;ms-source-network=federation The Frontend Server informs the organize site now that the connection was failing and Edge Server starts it proxying process. TL_INFO(TF_PROTOCOL) [0]097C.0C74::07/11/2014-11:15:26.299.00000507 (SIPStack,SIPAdminLog::ProtocolRecord::Flush:ProtocolRecord.cpp(265))[3798769121] $$begin_recordTrace-Correlation-Id: 3798769121 Instance-Id: BF9DF Direction: incoming;source="internal edge";destination="external edge" Peer: LYNCFEPOOL01.INTERNAL.AD:51714 Message-Type: request Start-Line: ACK sip:[email protected];gruu;opaque=app:conf:focus:id:V3JZ92CZ SIP/2.0 From: "Caller, Nils"<sip:[email protected]>;tag=e4776a37ed;epid=f5710ea2b3 To: <sip:[email protected];gruu;opaque=app:conf:focus:id:V3JZ92CZ>;tag=EDEE8C0427072C271B9B823E3B26BC5F Call-ID: 53fa037467934a3aa58afa7da405cffd CSeq: 1 ACK Via: SIP/2.0/TLS 10.10.10.127:51714;branch=z9hG4bKDFE93E20.E0C27AFE227343AD;branched=FALSE Max-Forwards: 70 Content-Length: 0 ms-diagnostics-public: 5012;reason="ACK is being generated on receipt of a failure final response for an INVITE forked by application";AppUri="http%3A%2F%2Fwww.microsoft.com%2FLCS%2FUdcAgent" Processing the ACK so it can be send to the organizer TL_INFO(TF_DIAG) [0]097C.0C74::07/11/2014-11:15:26.299.00000637 (SIPStack,SIPAdminLog::WriteDiagnosticEvent:SIPAdminLog.cpp(802))[3798769121] $$begin_recordSeverity: information Text: The message has a Discovered Domain SIP-Start-Line: ACK sip:[email protected];gruu;opaque=app:conf:focus:id:V3JZ92CZ SIP/2.0 SIP-Call-ID: 53fa037467934a3aa58afa7da405cffd SIP-CSeq: 1 ACK Peer: sip.singleip.com:5061 Data: domain="singleip.com" Processing and check against the discovered domain list. TL_INFO(TF_DIAG) [0]097C.0C74::07/11/2014-11:15:26.299.00000679 (SIPStack,SIPAdminLog::WriteDiagnosticEvent:SIPAdminLog.cpp(802))[3798769121] $$begin_recordSeverity: information Text: Routed a request to a Discovered Domain federated peer SIP-Start-Line: ACK sip:[email protected];gruu;opaque=app:conf:focus:id:V3JZ92CZ SIP/2.0 SIP-Call-ID: 53fa037467934a3aa58afa7da405cffd SIP-CSeq: 1 ACK Peer: sip.singleip.com:5061 The ACK is now send the sip.singleip.com organizer site. TL_INFO(TF_PROTOCOL) [0]097C.0C74::07/11/2014-11:15:26.299.00000685 (SIPStack,SIPAdminLog::ProtocolRecord::Flush:ProtocolRecord.cpp(265))[3798769121] $$begin_recordTrace-Correlation-Id: 3798769121 Instance-Id: BF9DF Direction: outgoing;source="internal edge";destination="external edge" Peer: sip.singleip.com:5061 Message-Type: request Start-Line: ACK sip:[email protected];gruu;opaque=app:conf:focus:id:V3JZ92CZ SIP/2.0 From: "Caller, Nils"<sip:[email protected]>;tag=e4776a37ed;epid=f5710ea2b3 To: <sip:[email protected];gruu;opaque=app:conf:focus:id:V3JZ92CZ>;tag=EDEE8C0427072C271B9B823E3B26BC5F Call-ID: 53fa037467934a3aa58afa7da405cffd CSeq: 1 ACK Via: SIP/2.0/TLS 10.11.10.84:61621;branch=z9hG4bK1616E949.64036B07705F839E;branched=FALSE Via: SIP/2.0/TLS 10.10.10.127:51714;branch=z9hG4bKDFE93E20.E0C27AFE227343AD;branched=FALSE;ms-received- port=51714;ms-received-cid=4B8F00 Max-Forwards: 69 Content-Length: 0 ms-diagnostics-public: 5012;reason="ACK is being generated on receipt of a failure final response for an INVITE forked by application";AppUri="http%3A%2F%2Fwww.microsoft.com%2FLCS%2FUdcAgent" Client doesn’t open Lync when meeting link is clicked. In some circumstance you will experience an issue join the conference with your client. It is important verifying the file association with will open the local installed client. Test Web App and open an Internet Explorer forcing the Conference to take place inside the browser. Copy the Join URL from the meeting invite, and then paste it into Internet Explorer. (Warning: Don't press Enter yet.) Add "?sl=1" to the end of the URL, and then press Enter. Validating Conference Settings and Expiration Conferencing is controlled via policies and global settings. The policy will control the behavior and the permitted features a user can use in a conference. Set-CsConferencingConfiguration: Beside a Content Grace Period, the time after a conference is retired when last activation occurred (someone joint). A reoccurring meeting also follows this principle if an end date was set. NOTE: Some disallowance will not be proper announced to the end user/ client. Meaning if a feature is not available, the conference will fail and the user is informed contacting her/his administrator. You need first to validate if the user was permitted for this action or denied by an assigned policy. Activation and Deactivation Before a user didn’t join a conference, it is not activated. After the activation, the Focus (not MCU factory) will check settings and permission within the backend database. From here the process starts, where the Focus gets in touch with the MCU Factory looking for getting details of available Conferencing Servers, than the Focus starts building those Conferencing Servers for its conference. You can have a look into the databases as well, finding those information soon the conference is activated, that information is placed into the RTCLocal | RTCDYN | ActiveConference database. Thanks of Richard, here explored more database fields, so you are now enabled reading those information and utilize the value for your support case. As well he explained perfectly the individual components involved when a conference was started.  ConfID: The conference ID is important to note, as it’s a primary key to other tables.  ConfStateVersion: The ConfStateVersion is a counter of changes occurring in your meeting.  Locked: This is a bit field and tells us if the meeting is locked (True – 1). A locked meeting will not allow any new participants.  AdmissionType: A TinyInt field with a few different options. o 3 – Anyone (No Restrictions) o 2 – Anyone from my organization or the meeting organizer. o 1 – People I Invite  AutoPromote: Another TinyInt field. o 0 – People scheduled as presenters o 1 – Anyone from my organization o 2 – Anyone (No Restrictions)  PstnLobbyBypass: Exactly what it sounds like. If set to True (1) than PSTN users get into meetings directly.  LastPartID: Not 100% sure what the role of this field is.  LastEnterprisePartLeaveTime: Date and time of the last authenticated users to leave a meeting. This is important later on.  ActivationInstance: GUID used by the system.  IsLargeMeeting: True (1) or False (0) if it’s a large meeting. The moral of this section is. Once a meeting is joined, than we create an instance in the database for an active conference. There are several moving parts to the creation of a conference. Focus is a SIP endpoint that represents the actual conference in the system. It’s job is a central gatekeeper. It’s pretty much responsible for everything for the conference. From authentication, requesting conferencing servers, etc. Focus Factory handles the logical creation of deletion of conferences for scheduled meetings in the database. Conferencing Server Factory determines the availability and health of the Conferencing Servers in the environment. During the meeting creation process, it’s responsible for telling the Focus which servers to place which modalities on. Now having a look into the defined possibilities for conference deactivation. First, deactivation and expiry are two different events. The deactivation refers to action of tearing down a particular instance of a conference, a job for the Focus to be monitored with. The deactivation can be either manual or automatic. Manually, three way are existing. Either the presenter clicks the “End Meeting” action, which force all participant to leave the meeting. Another action can be activated by “deleting the meeting from Outlook”. Here the focus will instantly deletes the meeting. User which were joint are disconnected. But it will not delete the users default meeting space. Last but not least, the user is removed from Lync/ Skype for Business. This triggers an automated process deleting the users active conference at the same time. Automatically, there are another three way. First, if all users have left the conference. Which is most likely happened. For around another 20min the Conferencing Announcement Service (CAS) will stay in this conference, so it stays in the background. Next possible deactivation is after 90min. At this point in time, the Focus will terminate the conference if no enterprise user joined the meeting or if all of them have left. (Federated and PSTN/Anonymous user are not subject to count). Richard tracked the related SIP BYE message sent to the CAS. The last deactivation rule is after 24hrs, meaning the no one joint. Resetting a default Conferencing ID I urge you reading the full article from Richard related to this support topic. http://masteringlync.com/2013/10/10/resetting-default-conference-ids/ Therefore I summarize the relevance for database information if you need supporting conference from this prospective. If you invite other into e meeting, you can either copy or email a link, which include a unique ID which is used to identify the conference which should be joined. In some cases it might be required to change or even analyze this user associated ID. This could also be happened e.g. after SIP Domain change. Please not, if only a small number of users need to change their conferencing ID, let them better do so via the Dial-In web page. Lync 2013 and Skype for Business have the conferences stored locally on the front-end servers, they are found in the RTClocal database instance. (not on the SQL Server back-end). In this instance you will find three database, our database from interest is the RTC, where have two tables REOURCES and CONFERENCE. The first table contains the user loggedin to the this particular front-end server (internal, external or even partners,…). From here you need to find your user and the related ReferenceID. You have to use a SQL query: SELECT * FROM resource WHERE userAtHost = ‘[email protected]’ Here we can see 293 is Richards resource ID. So now if you got to the conference table you can get back all of the conferences that are assigned to him: The conference with the value: TRUS in the STATIC field is the default conferencing ID. For you support, all other conferences can be identified as FALSE. The default Conferencing ID is e.g. used by Outlook, when creating a meeting e-mail and the meeting plugin is communicating with the front-end server, which the starts a lookup. If the value is TRUE, this ConferenceID will be returned to the user. If no TRUE value could be found, a new ConferenceID is generated and marked a STATIC (default). The next screenshots represent the process of a newly generated default ConferenceID: Now a new ID is generated. Richard than tested the different behavior in Outlook: Outlook Client Open (User has not yet requested a meeting today) The outlook client will reach into the database, find there is no static ID and returns a new static ID to the user by generating a new record. Outlook Client Open (User has requested a meeting today) The outlook client will use the cached information and schedule the meeting using the old ID. What happens when a person tries to join an existing (old) meeting? Since you have not deleted the old conference, the system will behave like normal and allow you into the conference. However, if you are doing this type of solution, most likely that old conference was broken and that is why you are doing this. What happens if you modify the meeting with the old static ID in Outlook? Outlook when opening the meeting will go and verify the state of the meeting. Since Outlook believes it’s the “default” meeting it will prompt you that things have changed: After clicking OK, the meeting will automatically update to the new static (default) conference and instruct the user to send an update to all participants. Once you have tested this, you could make the change using a simple SQL Update Query. Note: This would never be considered the supported solution but occasionally you have to go outside the box to fix a very broken system. Troubleshooting Lync and Skype for Business Web Services The essential security part from Lync and Skype for Business is the segregation of web services into internal and external IIS web pages. Therefore if one service or the entire web page will be compromised or crashed, not the entire system will run into issues. Manly on this particular service or web site. Certificate assignment is also essential and part of the deployment wizard. Internal and External Web Services IIS For troubleshooting it is recommended that you have full understanding of the different types of session establishment. The behavior for example between an IM Session and an A/V call is quite different. In case you need to support issues, it is essential to know where to identify and where to start with your support approach. Remote/ Internet Inside DMZ LAN SIP Proxy Registrar USER USER (Lync Edge) Lync Frontend 443 Reverse Proxy Office Web App (e.g. IIS ARR) Server (WAS/ WAC) In Lync and Skype for Business, several service are externally published via web services. We can differentiate those services into two categories, client and web page. The client handles service like address book downloads or expansion of distribution groups via the Reverse Proxy. While the Meeting Join and Web Application are services publish to non-Lync/ Skype for Business clients. Isolated from those service is the publication of Power Point rendering in Web Conferences. NOTE: For WAC deployment refer too: http://lyncuc.blogspot.de/2013/09/deploy-office-web-apps-server- 2013-and.html The Front-End IIS is segregated into two dedicated web sites, one for internal access (443) and one for external requests (4443): The IIS Web Services are listed in the picture below, each of the certificate provide several functions and is split into the “external” and “internal” web site. Having a look into the provided services:  To enable users to download files from the Address Book Service  To enable clients to obtain updates  To enable conferencing  To enable users to download meeting content  To enable users to expand distribution groups  To enable phone conferencing  To enable response group features  To enable mobile client features (see next chapter) In this extract, the following virtual directories are created and should never be changed manually. Only via the deployment wizard or management shell commands. The web site exists with the “Internal Web Service FQDN” and an exact copy for the “External Web Service FQDN”. On the external service the physical file location is changed to “ext” instead of “int”. Lync Web Address Description Service Address Book https://<Internal Location of Address Book Server Server FQDN>/ABS/int/Handler download files for internal users. Autodiscover https://<Internal FQDN>/Autodiscover Location of the Lync Server Service Autodiscover Service that locates mobility resources for internal mobile device users. Client updates http://<Internal FQDN>/AutoUpdate/Int Location of update files for internal computer-based clients. Conf http://<Internal FQDN>/Conf/Int Location of conferencing resources for internal users. Device updates http://<Internal Location of unified FQDN>/DeviceUpdateFiles_Int communications (UC) device update files for internal UC devices. Meeting http://<Internal FQDN>/etc/place/null Location of meeting content for internal users. Mobility Service https://<Internal FQDN>/Mcx Location of Mobility Service resources for internal mobile device users. (Lync 2010) Mobility Service https://<Internal FQDN>/UCWA Location of Mobility Service resources for internal mobile device users. (Lync 2013 and Skype for Business) Group http://<Internal Location of the Web service that Expansion and FQDN>/GroupExpansion/int/service.asmx enables group expansion for Address Book internal users. Also, the location of Web Query the Address Book Web Query service service that provides global address list information to internal Lync Mobile Microsoft Lync 2010 Mobile clients. Phone http://<Internal Location of phone conferencing Conferencing FQDN>/PhoneConferencing/Int data for internal users. Device updates http://<Internal FQDN>/RequestHandler Location of the Device Update Web service Request Handler that enables internal UC devices to upload logs and check for updates. Response Group http://<Internal FQDN>/RgsConfig Location of Response Group application Configuration http://<Internal FQDN>/RgsClients Mobility Services (for mobile clients) There are different scenarios where a mobile 2013/ Skype4Business client can establish its connection. First, we are not making use of MCX (virtual Directory) anymore. The actual client use UCWA which was introduced with Lync 2013 CU February 2013 and is still valid for Skype for Business. Please refer to Microsoft Technet and Jeff Schertz blog. The three possible scenarios are: 1. Internal Mobile Client establish a connection to an internal Client 2. Internal Mobile Client establish a connection to an internal Client, but cannot connect to the internal client due to a firewall blocking. But has Internet connectivity 3. Internal Mobile Client establish a connection to an external Client (REMOTE) This scenarios are important for troubleshooting. While you did your planning’s for Mobile Services, you need to know how the network related setup will be. NOTE: Although mobile applications can also connect to other Lync Server 2013 services, the requirement to send all mobile application web requests to the same external web fully qualified domain name (FQDN) applies only to the Lync Server 2013 Mobility Service. Other mobility services do not require this configuration. Illustration about generic setup: SIP Domain: customer.com Lync Server 2013 Pool INTERNAL WEB SERVICE FQDN (VIP) EXTERNAL WEB SERVICE FQDN (VIP) 1. Query LYNCDISCOVERINTERNAL 2. Query LYNCDISCOVER than AutoDiscover provides MOBILITY URL (Ext Web Service Listener IP: IIS (Lync Web Components) FQDN) 202.x.x.x HTTPS://mobility URL, Revers Proxy External WebSite (4443) extweb.customer.com Mobility Service (MCX/ UCWA) Autodiscover Service HTTPS GET LYNCDISCOVER.customer.com Internal WebSite (443) Mobility Service (exits, but not activated) Autodiscover Service DNS ZONE: customer.com Proxy Lyncdiscover A 202.x.x.x HTTPS GET LYNCDISCOVERINTERNAL.customer.com Extweb A 202.x.x.y HTTPS://mobility URL, extweb.customer.com DNS ZONE: customer.com INTERNET DMZ INTERNAL/ LAN Lyncdiscover A 10.z.z.z Extweb A 202.x.x.y HTTPS:// MOBILITY URL EXTERNAL WEB SERVICE FQDN Having a look into the three scenarios and see where along the signaling and media path is established. At the first very beginning of a mobile client login stands the autodiscovery. The first DNS query is against the lyncdiscoverinternal and the second query is against the external lyncdiscover. Via the Reverse Proxy Server, the discover XML of the “link token” will be submitted to the mobile client. Generally the software is hardcoded and is making use of <Link token="Ucwa". It is therefore required that a mobile client must be able to discover the correct URL. In your troubleshooting process, once again the validation of core network services is essential. It needs to be understood, that Lync/ Skype for Business mobile clients make use of hard coded virtual directories. First, Lync 2010 clients utilize the MCX directory, which can be tested with a specific Test-CS command. All newer clients are hard coded to the UCWA feature. Even if we see later, that UCWA exists internally and cannot be chosen for new clients, this is a requirement. If you would be able using the internal service provided (point the DNS internally) the mobile device must trust the issued web certificate. Which is unlikely to be happened with BYOD deployments and this is quite a hassle. Now we are having a look into the dedicated scenarios first. Scenario 1 (internal mobile/internal full client): Inside DMZ LAN signaling signaling 2 LYNC signaling Autodiscover 1 3 SIP Proxy Registrar Media (Lync Edge) Lync Frontend Reverse Proxy (e.g. IIS ARR) Internal User The mobile client is discovering the internal LYNCDISCOVERINTERNAL URL (1) and will make use of the of the EXTERNAL MOBILITY URL (FQDN - “link token=UCWA”) (2). Different is the media establishment, the client provide the candidates and are entitled for a direct peer-to-peer setup (3). Important is the network path and it must be non NATed, a direct route. Scenario 2 (internal mobile behind internal firewall/internal full client): Remote/ Internet Inside DMZ LAN signaling signaling signaling Media Media LYNC 1 signaling 2 Autodiscover Media SIP Proxy Registrar (Lync Edge) Lync Frontend 3 Media Reverse Proxy (e.g. IIS ARR) Internal User A usual deployment for mobile devices (or BYOD – Bring Your Own Device) is a deployment in a dedicated e.g. WiFi network secured with a firewall. The autodiscovery process is identically to the first scenario. Consider, if the network, where the device is placed, in is not able, at any point of time. To connect to the internal services, it must be threaded as external! If separated mobile device is in this scenario is unable to negotiate a direct media path (1), due to a firewall, port closing or filtering, the mobile client must rely on the Edge Server and has to tunnel the signaling/ media. The mobile device will connect to and send its media session to the external Edge interface (2). The internal full client follows the standard connection process. In this example the full client must connect media to Edge Server internal interface. Scenario 3 (internal mobile/external full client): Remote/ Internet Inside DMZ LAN Media Media signaling signaling LYNC signaling Autodiscover signaling Reverse Proxy Registrar (e.g. IIS ARR) Lync Frontend 1 Media Media 2 External SIP Proxy User (Lync Edge) This scenario is identically with the scenario 2. Nevertheless, the difference is that the call to the external full client is rerouted via Edge Server and send to the external side again. First to the external Edge interface (1) than back through the Edge server to the remote client (2). Having a look into the discovery and logon process: Generally the first step is the Autodiscovery process, where a client hard coded query first the LyncDiscoverInternal FQDN and if this fails, it queries the LyncDiscover FQDN. Next step is to analyze the XML it provides: What we can see here is the complete list of all URL possible for any kind of query. Every application can now choose their own required URL. Since we are focusing on the mobility services, we have to identify the MCX and the UCWA directories only. The authentication requires a Web Ticket for the entire communication, it is request and authenticated with NTLM. The Web Session Ticket is valid for 8 hrs. Authentication to Exchange provided services require no Web Ticket and use NTLM every time a query is initiated. Lync 2010 Mobile App: All Lync 2010 Apps are only using the external web service FQDN connecting to the MCX mobility services. This makes it more clear why the DNS and is related routing must be associated with the drawing from above. It has the exact same value: Internal MCX service : https://lyncwebext1.xiopia.com/Mcx/McxService.svc External MCX service : https://lyncwebext1.xiopia.com/Mcx/McxService.svc Lync 2013/ Skype for Business Mobile App (Windows, iPhone, iPad and Android) The actual app is a bit trickier to understand. If we have a look into the discovery URLs provided, we will find: Internal UCWA service : https://lyncwebint1.xiopia.local/ucwa/v1/applications External UCWA service : https://lyncwebext1.xiopia.com/ucwa/v1/applications Theoretically, we could assume that mobile clients could utilize those URLs. But indeed they aren’t used and only reserved for future use or other 3rd party apps. (It makes sense for those 3rd party apps, where those app leverage on the internal/ external FQDN). Lync 2013 mobile client and the new Skype for Business client leverage on a THIRD URL provided: Named UCWA. We find this URL if we switch to the Lync Connectivity Analyzer Detailed View: There the information are more detailed: AccessLocation="External"> <User><SipServerInternalAccess fqdn="lyncpool1.xiopia.local" port="5061" /> <SipClientInternalAccess fqdn="lyncpool1.xiopia.local" port="5061" /> <SipServerExternalAccess fqdn="sip.xiopia.com" port="5061" /> <SipClientExternalAccess fqdn="sip.xiopia.com" port="5061" /> <Link token="Internal/Autodiscover" href="https://lyncwebint1.xiopia.local/Autodiscover/AutodiscoverService.svc/root" /> <Link token="Internal/AuthBroker" href="https://lyncwebint1.xiopia.local/Reach/sip.svc" /> <Link token="Internal/WebScheduler" href="https://lyncwebint1.xiopia.local/Scheduler" /> <Link token="Internal/CertProvisioning" href="https://lyncwebint1.xiopia.local/CertProv/CertProvisioningService.svc" /> <Link token="External/Autodiscover" href="https://lyncwebext1.xiopia.com/Autodiscover/AutodiscoverService.svc/root" /> <Link token="External/AuthBroker" href="https://lyncwebext1.xiopia.com/Reach/sip.svc" /> <Link token="External/WebScheduler" href="https://lyncwebext1.xiopia.com/Scheduler" /> <Link token="External/CertProvisioning" href="https://lyncwebext1.xiopia.com/CertProv/CertProvisioningService.svc" /> <Link token="Internal/Mcx" href="https://lyncwebext1.xiopia.com/Mcx/McxService.svc" /> <Link token="External/Mcx" href="https://lyncwebext1.xiopia.com/Mcx/McxService.svc" /> <Link token="Ucwa" href="https://lyncwebext1.xiopia.com/ucwa/v1/applications" /> <Link token="Internal/Ucwa" href="https://lyncwebint1.xiopia.local/ucwa/v1/applications" /> <Link token="External/Ucwa" href="https://lyncwebext1.xiopia.com/ucwa/v1/applications" /> <Link token="External/XFrame" href="https://lyncwebext1.xiopia.com/Autodiscover/XFrame/XFrame.html" /> <Link token="Internal/XFrame" href="https://lyncwebint1.xiopia.local/Autodiscover/XFrame/XFrame.html" /> <Link token="XFrame" href="https://lyncwebext1.xiopia.com/Autodiscover/XFrame/XFrame.html" /> <Link token="Self" href="https://lyncwebext1.xiopia.com/Autodiscover/AutodiscoverService.svc/root/user" /> </User></AutodiscoverResponse> We can identify that the URL is the same as the External/UCWA. This is why the internal DNS for this SIP Domain providing the Web Services must point with the External Web Service FQDN to the Reverse Proxy. Address Book Web Services for Mobile Devices The mobile client can download only a few lists compared with the full client. The downloadable lists are the buddy list and normalization rules (for making calls). Different from the full client is the address book, since the AB can become quite large, the mobile client makes use of the Address Book Web Services. This requires that for all search requests to internal Lync enabled users is made via a web based query (ASWQ). By default only the Lync Phone Edition, Web App, and Mobile clients will leverage ABWQ based searches against the Lync Server’s rtcab (or rtcab1) database which stores the same address book information that the ABS server files do. Before I dig deeper it is important to remember once more, Lync and Skype for Business rely on phone numbers in the E.164 format. If a user cannot be found, this is mainly due to a wrong number format. (Where the address book is stored on the server, you will also find two additional files, one with a normalization patter and another file listing all users which can’t be normalized and are excluded from the AB). You can trace the ABS activities with the OCSLogger: Especially for mobile client the test command is essential: Test-CsAddressBookWebQuery -TargetUri https://atl-cs- 001.litwareinc.com/groupexpansion -UserSipAddress "sip:[email protected]" - TargetSipAddress sip:[email protected] -external Important is the parameter “-external”, this enables the test against the external web services, used by mobile clients. You might also use the Get-Credential command for authentication. The parameter –Target Uri and TargetFqdn cannot be use simultaneously. If you test the ABWQ, the TargetFwdn is required. One unvalidated issue on iOS can be that the user’s mobile number was not provided and therefore the wired behaviors are happened. Else you should check the msExchHideFromAddressLists parameter, which also has an impact to Lync/ Skype for Business full clients. At last, I’m often asked if you can exclude users from the address book. Well you can. You have to use the ResKit utility ABS Configuration Tool and define an e.g. AD Attribute Name and check the “Exclude all AD User who have…” option. Viewing policy setting in Lync/ Skype for Business, user the following command: Get-CsClientPolicy | Select-Object Identity,AddressBookAvailability | ft It is providing the client setting if Web Search and or File Download is enabled. If a mobile client can query this AB, it might also be happened the Web Search is disabled. An indicator can also be if you don’t find or see “old” GALcontacts.db and GALcontacts.db.idx files on the full client. Then, if you made use of the msRTCSIP-GroupingID, also grouped and therefore incorrect search results might occur. Troubleshooting Office Web App Server Lync 2013 Server will identify the internal and external URL configured with the WAC Server. Now we need a verification, that Lync 2013 Frontend has the correct setting. Filter the Lync FE EventLog for all WAC related events: 41032 and 41034 You will find an entry similar like this: - System - Provider [ Name] LS Data MCU - EventID 41032 [ Qualifiers] 17402 Level 4 Task 1018 Keywords 0x80000000000000 - TimeCreated [ SystemTime] 2013-09-04T11:33:32.000000000Z EventRecordID 5473 Channel Lync Server Computer WACinternal.domain.intern Security - EventData https://WACinternal.domain.intern/m/Presenter.aspx?a=0&e=true& https://WACinternal.domain.intern/m/ParticipantFrame.aspx?a=0&e=true& https://webapp.extDomain.de/m/Presenter.aspx?a=0&e=true& https://webapp.extDomain.de/m/ParticipantFrame.aspx?a=0&e=true& If a client joined a conference and need to receive the Power Point presentation, a SERVICE SIP messages is submitted to the client containing the reference URL to the Office Web App Server. (“marked in red”): 09/04/2013|14:55:10.399 558:61C INFO :: SERVICE sip:[email protected] SIP/2.0 Via: SIP/2.0/TLS 192.168.1.105:52102 Max-Forwards: 70 From: <sip:[email protected]>;tag=1216ee8c42;epid=fe5337abb5 To: <sip:[email protected]> Call-ID: c858fcb8e8dd4390b20bd3957050e6d8 CSeq: 1 SERVICE Contact: <sip:[email protected];opaque=user:epid:qxOEj3bU1VaO18cHg7Lu4wAA;gruu> User-Agent: UCCAPI/15.0.4517.1004 OC/15.0.4517.1004 (Microsoft Lync) Proxy-Authorization: TLS-DSK qop="auth", realm="SIP Communications Service", opaque="0A6C31A1", targetname="lyncserverppol.domain-a.com", crand="f0cb3d02", cnum="276", response="1ccdd5bb003db213989aeda53ed2f12c6e7d97ce" Content-Type: application/msrtc-reporterror+xml Content-Length: 1177 <reportError xmlns="http://schemas.microsoft.com/2006/09/sip/error-reporting"><error toUri="sip:[email protected];gruu;opaque=app:conf:focus:id:TYQF4ZHC" callId="3a63424bce4f4542a1878cf29782fd35" fromTag="6eec3407d5" toTag="23480080" requestType="" contentType="" responseCode="0"><diagHeader>54025;reason="A viewing URL navigation was attempted."; ClientType=Lync;Build=15.0.4517.1004; ContentMCU="sip:[email protected];gruu;opaque=app:conf:data- conf:id:TYQF4ZHC";ConferenceUri="sip:thomas.poett@domain- a.com;gruu;opaque=app:conf:focus:id:TYQF4ZHC";LocalFqdn="lyncserver01.domain-a.com"; Url="https://webapp.domain- a.com/m/ParticipantFrame.aspx?a=0&amp;e=true&amp;WopiSrc=https%3A%2F%2Fmgacsap4 0.domain-a.com.intern%2FDataCollabWeb%2Fwopi%2Ffiles%2F5-1- 2EB85D8&amp;access_token=AAMFEHCysGizzW9ZqKYwzMlxwFQGEM34svWrZyP- zsPbJWGjNzKBEHCysGizzW9ZqKYwzMlxwFSCAtO2gyAQW9O14tatIkg7- CY3o087igqpE1IlNxyRe8SIPyn0bYYI1bAhMch30AgIDURhdGFDb2xsYWJXZWI&amp;&lt;fs=FULLSC REEN&amp;&gt;&lt;rec=RECORDING&amp;&gt;&lt;thm=THEME_ID&amp;&gt;&lt;ui=UI_LLCC&amp; &gt;&lt;rs=DC_LLCC&amp;&gt;&lt;na=DISABLE_ASYNC&amp;&gt;"</diagHeader><progressReports/ ></error></reportError> Troubleshooting: Attempted Office Web Apps Server discovery Url: https://webapps.extDomain.de/hosting/discovery/ If you receive a similar XML extract, the Office Web App Server is working fine, if any other issue is presented troubleshoot the configuration. Received error message: The remote certificate is invalid according to the validation procedure. The number of retries: 13327, since 2/27/2013 9:07:42 PM. Or Lync 2013 PowerPoint sharing issue: “There was a problem verifying the certificate from the server. Please contact your support team.” CERTUTIL –URLFETCH –VERIFY “OfficeWebApp.cer” Use this command to verify if the CDP for CRL checkup is correct. This verifies the HTTP connection. ERROR: IIS Error 500.21 For Windows Server 2008 R2 %systemroot%\Microsoft.NET\Framework64\v4.0.30319\aspnet_regiis.exe -iru iisreset /restart /noforce For Windows Server 2012 dism /online /enable-feature /featurename:IIS-ASPNET45 Another issue is with WAS installed on Windows Server 2008. There is one hotfix which must be applied to the OS: If Windows Server 2008 R2 reports: KB2592525 is not applicable for your computer, you need to remove the conflicting Update: KB2670838 Very often you will find a typo in the Lync Topology, where the discovery URL was typed in wrongly. Also verify the correct address here too. Issue with converting a PPTX file during upload: "[File Name].pptx can't be converted for presentation because PowerPoint is not installed. Please install PowerPoint and try again." To fix the problem you will need to edit the [HKEY_CLASSES_ROOT\TypeLib\{91493440-5A91-11CF-8700- 00AA0060263B}\2.b\0\Win32] @="C:\\Program Files\\Microsoft Office 15\\Root\\Office15\\MSPPT.OLB" and point it to the Office 14/15 directory in regedit. The easiest way to do it would be to copy the registry value for the TypeLib 2.a and paste it over the 2.b value. Enterprise Voice The article Update Version 2.0 will contain more about Enterprise Voice. Voice Route and Trunk parameter With Lync 2013 the improvements regarding Enterprise Voice were driven more towards an Enterprise capable system. Therefor it’s not surprising we see some differences in Trunk Configurations too. I focus now only on the features visible in the Lync Control Panel (CSCP). First we need to determine what type of Trunk Configuration we need: Pool or Site  Pool (Site): assigned to a Lync Site defined in the Topology  Site (Service): a service, like PstnGateway object defined in the Topology Maximum early dialog supported: maximum count of INVITE dialog (* see detailed description) Encryption support level : (SRTPMode) – define if media traffic is encrypted or not Enable Media Bypass : define if the Mediation Server can be bypassed by the PSTN connection point and the client Centralized media processing : if the Gateway object supports an unique IP for signaling and media traffic Enable refer support : SIP REFER command support for Call Transfer (RFC3515) Enable RTP latching : This parameter will enabled Media Bypass option for Client (RTP/ RTCP) located behind NAT or Firewall. The SBC must support latching. Enable forward call history : Call history data can be forward to the trunk. Enable forward P-Asserted-Identity data : (P-Asserted-Identity (PAI) header can be forwarded along the call to provide a way the caller can be identified. Enable outbound routing failover timer : If call were not answered from the associated gateways after 10 sec, the call will be forwarded to the next available trunk, else if no additional trunks, a call drop occurs. Associated PSTN Usage : As described while I explained the Voice Route, PSTN Usage records are required to be configured with this Trunk too. Associated translation rules: Translations rules modifying the outgoing call Calling number translation rules : Will modify the calling number (person who called) Called number translation rules : modify the called number (person being called) *) See the chapter above for detailed explanation for calling vs. called There are many more option which can be configured on Trunk Configuration in Lync 2013, like the c3p, Office 365 Online Voice, E-9-1-1 (Presence Information Data Format Location Object : PIDF-LO) and much more. This will be part in one of my next Blogs, when I’m talking about Deep-Inside Enterprise Voice. *) Early Dialogs: RFC 3261: A dialog contains certain pieces of state needed for further message transmissions within the dialog. This state consists of the dialog ID, a local sequence number (used to order requests from the UA to its peer), a remote sequence number (used to order requests from its peer to the UA), a local URI, a remote URI, remote target, a boolean flag called "secure", and a route set, which is an ordered list of URIs. The route set is the list of servers that need to be traversed to send a request to the peer. A dialog can also be in the "early" state, which occurs when it is created with a provisional response, and then transition to the "confirmed" state when a 2xx final response arrives. For other responses, or if no response arrives at all on that dialog, the early dialog terminates. In other words, SIP Messages are part of a communication (dialogs), e.g. in our Trunk Configuration negotiation about the inside protocols. We define here how many INVITES can be negotiated. Some of the SIP Trunk Provider support less than the default setting in Lync, we need therefor a Trunk Configuration to support the SBC requirements given to us. References http://lyncuc.blogspot.de/2014/04/internal-certificate-deployment-in-lync.html http://lyncuc.blogspot.de/2013/02/demystify-lync-enterprise-voice-phone.html http://kemptechnologies.com/files/assets/documentation/7.1/technical-notes/Technical_Note- MS_Lync_2013_Server_Security_Guide.pdf http://en.wikipedia.org/wiki/Transmission_Control_Protocol http://en.wikipedia.org/wiki/User_Datagram_Protocol Other blogs and references: Special thanks is going to my other fellow Lync MVPs for inspiring me writing this Troubleshooting Guide and they provided the most valuable information’s to me, which made quite a part of this guide. Jeff Scherz: http://blog.schertz.name Richard Brynteson: http://masteringlync.com Justin Morris http://www.justin-morris.net https://channel9.msdn.com/Events/Speakers/Thomas-Binder https://channel9.msdn.com/Events/Speakers/Thomas-Poett RFC’s: SIP Protocol: https://tools.ietf.org/html/rfc3261