How to Install Autotraces

This guide walks you through the steps required to install Autotraces, custom applications packaged into a Dialplan which simplify the process of log/data collection. 

Created: July 2021

Updated: October 2021

Permalink: https://wildix.atlassian.net/wiki/x/rhjOAQ

Purpose of Using Autotraces

The purpose of using Autotraces is to make log/data collection simple. 

Quite often, Wildix Global Support needs data about an incident reported to them via the support process to determine the cause and suggest a fix or escalate an issue to the development team. Historically, this has sometimes been more difficult than it would seem on the surface, especially in cases where the issue is not easily reproducible. 

For this reason, the support team at Wildix created a series of custom applications that are packaged into a Dialplan. These custom applications will provide the following functionality:

  • Start logging that runs in the background
  • Allow a user/admin to report an incident via dialing a specified code in the Dialplans
  • Create a package of logs that are downloadable from the Backup system tab of WMS
  • Restart the logging after the creation of a package of logs to ensure that further reports can be made
  • Allow the admin to start, stop or change to the type of SIP traces to run in the background 

And, all of this can now be done from WMS or via dialing a specified code from a user on the system. No access or work in the Linux environment is required. 

Autotraces Installation 

Installing Autotraces is an easy process: 

  1. Import the Autotraces Dialplan 
  2. Upload the sounds file package required by that Dialplan 
  3. Set one or more numbers to the Dialplan 
  4. Call the number you put in the Dialplan for the first time to start the traces

See these steps in detail below.  

Step 1. Import the Autotraces Dialplan

  1. Download the Dialplan Autotrace_vXX.bkp (where XX is the version) from here: https://drive.google.com/drive/folders/1J2KSqodBPDcBw1uH3SptPbRT83SJfwuK 
  2. Once downloaded, go to the Dialplan rules in WMS, click the Import button and select the Autotraces Dialplan that you have downloaded to your computer

See screenshots below:


Step 2. Upload the sounds file package required by the Dialplan

  1. Download the sounds file package Autotrace_vXX_sounds.tar.gz (where XX is the version) from here: https://drive.google.com/drive/folders/1J2KSqodBPDcBw1uH3SptPbRT83SJfwuK
  2. Upload the backup file in the Backup system tab of WMS as shown below. First, click Choose File and upload it: 




  3. Now that the file is uploaded, click Upload


  4. Once it’s uploaded to the system, select the backup file and click Apply backup: 

  5. You will get a pop-up similar to the one below depending on your system. Do not worry about the restore network option. There are no network settings or any other configurations in this file. There are only sound files. So, simply click Yes to confirm that you want to apply the sounds package:


Step 3. Set one or more numbers in the Dialplan

  1. Make an entry in one or more Dialplans to make the Autotraces available via a pilot number. In the case below, an entry for 2222 is added into the "users" Dialplan, so users assigned to that Dialplan are able to call this pilot number to report an incident:



Note: Use the Set part of the Jump to Dialplan application to in_admin (a single underscore).

There are 2 underscore characters before the tracemail and host variable names:

  1. __tracemail is set to an email address of the person that you want to receive an alert about a reported incident. Example: partners tech support person’s email address
  2. __host should be set to the PBX name that you are installing Autotraces on. This information is then included in the email alert so that the person knows what system the incident report was generated on

Step 4. Call the number you put in the Dialplan to start the traces

  1. Log in as a user on the system (Collaboration, mobile app, provisioned phone, etc)
  2. Dial the pilot number that you built
  3. Follow the prompts to start the trace in whichever mode you choose. More information on trace types can be found in the More Information section

Usage 

After the initial setup, the trace will be running in whichever mode you started it in. At this point, you should consider who will be creating incident reports and what options you would like them to have when they dial the pilot number. If you do not want users calling this number to have the option to completely stop the trace from running or to change the trace type, you should edit the Set portion of the Jump to Dialplan application to be in_admin_user. See More Information section of this guide for more details. 

Use Case: 

  1. The admin has installed and started the Autotraces as described above
  2. The admin informs the end users of the choice of how to generate an incident report
  3. End user has some issue and calls the pilot number built
  4. User is asked to stay on the line to generate an incident report
  5. System asks the user to record a message indicating what the issue was
  6. The call to Autotraces ends with the calling party 
  7. The system stops the traces, creates a backup file of logs, and restarts the trace
  8. System emails the administrator information about the report (including recording)
  9. The administrator goes to Backup system tab of WMS on the PBX that the incident report was made and downloads the backup file that has all of the logs in it. See Contents of Trace section below for more information.  


Note: The person calling the pilot number to access Autotraces will get different prompts and possible actions to take depending on the value used to jump to the AA_autotrace Dialplan. See below for more information. 

Note: Do not stop the trace to create a log package. Instead, use the create incident report option. This option will create the backup file with all logs and automatically restart the trace so that another incident report can be made later.


More Information 


Important: The backup file has a .tar.gz file extension. This is a tar’d and zip’d file. You will need an unzip utility to untar and unzip this archive file (e.g. winzip, 7zip, etc). 

  • Places to point to when sending a call into the AA_autotrace Dialplan: 


Set Portion of Jump to

Allowed Control

in_admin 

Full control of all options. Caller can start a new trace, completely stop an existing one or even change the type of trace that is running.

Important: Stopping a trace does NOT create an incident report. To create an incident report, use option 7.

in_admin_user (suggested setting)

Start a trace if one is not started already, or create an incident report. User cannot change trace type or completely stop a trace.


  • Trace types to select from include the following: 


Trace Type

Content of the logs created

Standard

Pcap of SIP signaling 

Callweaver logs

Presence logs

WMS logs

Syslogs from phone (if configured in devices)

Audio

Same as standard mode plus RTP port range in pcap file (audio streams)

Extended

Same as audio mode but with additional rotated logs. More data but will use more disk space.

Custom

** for advanced use **

Admin can set a tcpdump command in the AA_autotrace Diaplan (entry 4)


Note: If syslog is desired from Wildix devices, set the syslog server of one or more devices as shown in the screenshot below. Once this is done, the device will send its syslog to the PBX and the syslog will be viewable in the pcap file(s) on port 514. 


Of course, replace NameOrIPofPBX with the actual value representing your PBX.

Contents of the Email Alert 

  • Subject: AUTOTRACE TYPEOFTRACE - Incident Report User Name (Extension) on PBX host.wildixin.com 
  • Incident reported on Date at Time 

-> Please download the logs from the menu Tools and utilities -> System backup

  • REMINDER: the calculation of the disk space is based on the creation of a single archive at a time, so remember to delete it after use. 

-> FORGETTING MAY CAUSE SERIOUS MALFUNCTIONING! 

  • Attachment: [user_recording.wav] -- Recording created by the user when they made the report. 

Note: In the above the following data description applies: 

  1. TYPEOFTRACE = the kind of trace that was running when the report was made
  2. User Name = full name on the user record in WMS for the extension that made the report
  3. Extension = the extension of the user record in WMS for the user that made the report
  4. Host.wildixin.com = value of the ‘host’ variable for the pilot number called to make the report


Contents of Trace 

Special Notes

Logs and pcap files are rotated by the system. This means that there often are multiple files with close to the same name except that they have some rotation number in them. 

Example: 

wms.log 

wms.log1.gz 

wms.log2.gz 

wms.log3.gz 

You can see by the date/time of the file which one will have content for what times. This is because the date/time of the file is the LAST time that the file was written to. 

Some files, including the backup file itself are compressed. You will need a utility to uncompress the files to have them be readable (e.g. winzip, 7zip, etc).

Contents of Different Trace Types

All trace types will include the information shown below. What is shown below is the content of a standard trace. 

Audio trace type will include RTP in their pcap files. 

Extended mode simply has more files (more data across a longer period of time). The custom mode will have all of the same data except that the included pcap files will include only the information related to your custom tcpdump command. 



  • Following the path for mnt, backups, traces, the content is as follows: 



Note: If the incident report was made via option 9 for issues regarding function keys or voicemail indicator issues, dump_classic.txt is replaced by dump_presence.txt. 

  • Following the path for var, log, the content is as follows: 


  • And, following further to the callweaver folder, the content is as follows:



  • Log file content is as follows:

File

Content Description

FILENAME.pcap

Pcap file captured by tcpdump. Open with Wireshark as standard pcap.

Caution: File extensions will include a number at the end: 0/1/2/3/etc. 

This is done by tcpdump in order to “rotate” capture files. This extension will need to be edited to be .pcap only for Wireshark to be able to open the file.

cw.log

Callweaver logs as you would see via terminal - pbxengine.

dump_classic.txt or dump_presence.txt

Significant amount of data about the state of the system at the time that the incident report was made. This includes system time, call group status, services uptime, presence information for users, etc.

syslog_device_YYYY-MM-DD.log

Syslog data from devices if configured in the Devices tab of WMS.

user_recording.wav

The .wav file of the the recorded information that the user created during the time that they created the incident report.

wms.log

wms.log.N.gz (where N is some number)

These files are wms.log files from the system. They are rotated over time and therefore you get a number and .gz extension (zip file). Note that the date/time of the file can help you to determine the file that has the the data of interest to you.

queue_log

queue_log_failed

Log files containing data about call group calls and users.