Linux Enterprise Node Troubleshooting Guide

  • 8 minute(s) read
Prev Next

Catchpoint Enterprise Node utility

Catchpoint Enterprise Node Instances come equipped with a utility that allows you to perform basic troubleshooting. This utility can be invoked by simply running catchpoint on the command line.

Basic Node Instance maintenance:

  1. Status of the Catchpoint Node services:
    $ catchpoint status
  2. Restart Catchpoint Node services:
    $ catchpoint restart
  3. Stop Catchpoint Node services:
    $ catchpoint stop
  4. Start Catchpoint Node services:
    $ catchpoint start

Log Files

Log files are extremely helpful when troubleshooting a Node. Everything that the Node does in logged into a log file. These log files are located at /var/3genlabs/hawk/syntheticnode/service/log location.

To see a list of all log files and their date and time stamp
$ ls -la /var/3genlabs/hawk/syntheticnode/service/log

Use the less command to open a log file (or any file) and display the information one page at a time.
$ less /var/3genlabs/hawk/syntheticnode/service/log/logfilename.log

Less command - tips & tricks

By default, the less command will scroll a single page at a time. You can change the number of lines that are scrolled when you press the space and f key by pressing the number immediately before pressing the key.

To make this number the default you can enter the number followed by the z key.

If you want to go back to the beginning of the output press lowercase g and to go to the end press uppercase G. To go to a specific line, enter a number before pressing the g or G keys.

You can search for text within the output using the / key followed by the text you wish to search or a regular expression.

If you have finished looking at a file you can load a new file into the less command by pressing the colon key : followed by the e or E key and the path to a file.

To exit the less command, press either the q or Q key.

Useful Command Line Switches

The following run time switches may be useful to you:

  1. less -bN - The N stands for a number of kilobytes to load into memory. By default, the value is 64 kilobytes, but you can specify any number you wish. If you enter -1 then the entire file will be loaded into memory which may or may not be a good idea depending on the size of the file.
  2. less -B - By default the less command allocates the required memory buffers by default when using piped output. You can use the -B switch to prevent auto buffering.
  3. less -c or less -C - By default the screen repaints by scrolling up the screen. To clear the screen from the top down use the -c or -C switches.
  4. less -e - Causes less to exit when it hits the end of the file for the second time
  5. less -E - Causes less to exit when it hits the end of the file for the first time
  6. less -f - Open special files such as directories using less
  7. less -F - Causes less to exit if a file is less than one screens worth of data
  8. less -g - Only highlight the last item found when searching
  9. less -G - Suppress highlighting altogether when searching
  10. less -hN - Specify the maximum number of lines the less command can scroll back
  11. less -i - Ignore case when searching unless uppercase characters are found in the search pattern
  12. less -I - Ignore case when searching
  13. less -jN - The N stands for a number. This determines where on the screen a line is placed when searched for. For example, searching for "hello world" will place the line found with "hello world" in it on line 1 if less -j1 is used.
  14. less -J - This displays a little asterisk in the left column (status column) which shows when a piece of text which you have searched for has been found.
  15. less -m - Displays the number of bytes through a file instead of a colon at the bottom of the screen
  16. less -M - Displays the line numbers of the output. For example, "lines 1-23"
  17. less -n - Suppress line numbers
  18. less -N - Display line numbers on each line
  19. less -o - This is used with piped output only. It outputs each page of the piped output to the file one page at a time. If the file exists it will ask whether you want to overwrite it.
  20. less -O - This is the same as -o except that it won't ask for confirmation before overwriting a file.
  21. less -p - This starts less at the first occurrence of the pattern specified.
  22. less -P “text” - This replaces the message at the bottom of the screen to the text specified
  23. less -q - This prevents the bell from buzzing when you reach the end of the file. Other reasons for the bell to ring such an invalid key press remain.
  24. less -Q - Suppresses all noises
  25. less -s - This condenses blank lines. For example, if a file has 4 consecutive blank lines and you use the less -s command only 1 blank line will be displayed.
  26. less -S - This causes long lines to be truncated rather than wrapping them onto the next line

Monitoring Log Files

Log files are generated at regular intervals so that the length of these log file is manageable. You can view the output of the service logs to STDOUT (similar to tail -f) by running:
$ catchpoint log

Gathering Log Files
At times you may need to get these log files for our Engineering team to look at, and there may be many of them. To collect these log files, create a tar file first:

$ catchpoint log --snapshot  
Copying cpsns log...   
Copying cprapi log...  
Copying service management log...Copying nginx log...   
Copying system log...   Compressing...  
Done. /var/tmp/catchpoint/{{hostname_timestamp}}.txz

The --snapshot or -s argument creates a snapshot of all logs related to Catchpoint Services and saves it to /var/tmp/catchpoint/log/. The output displays the complete path and the file name.

By default, the above command saves the log files from the last 24 hours. If you want to gather log files for more than 24 hours:

$ catchpoint log --snapshot -d 2   
Copying cpsns log...   
Copying cprapi log...   
Copying service management log...   
Copying nginx log...   
Copying system log...   
Compressing...   
Done. /var/tmp/catchpoint/{{hostname_timestamp}}.txz

The --days or -d creates a snapshot of all the logs related to Catchpoint Services for the # of days specified and saves it to /var/tmp/catchpoint/log/.

This argument is only valid when the option --snapshot or -s is used. The default value is 1.

Getting a TCPDump

Sometimes it useful to do a TCP dump to analyze what’s going on between the Enterprise Node and the internet and/or Catchpoint C&C. To perform a TCP dump:

$\> catchpoint packet-capture   
Hit Ctrl + C to stop.   
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes  
^C   
32 packets captured  
32 packets received by filter   
0 packets dropped by kernel   
Compressing...   
Done. /var/tmp/catchpoint/{{hostname_timestamp}}.cap.txz

The above command starts a packet capture and, when stopped, saves it as {{Hostname_Timestamp}}.cap in the /var/tmp/catchpoint folder. This .cap file can later be opened in a software like Wireshark for further analysis.

Command & Control Configurations

Every instance contains configuration settings that give us information as to what Command & Control servers it is communicating with. These details can be viewed by running:

$ catchpoint cc-config

Instance's Environment Variables

Once the service is installed and communicating with our Command & Control, it will be referring to some environment variables which reflect the settings the machine/appliance is set up with. For example, Proxy settings. To view these details:

$ catchpoint env

Updating Enterprise Node to the Latest Version

To update the Catchpoint Services to the latest version:

$ catchpoint update

Proxy Configuration

Users can now set up a proxy on a node Instance using the command line utility. Catchpoint supports Direct, Static & PAC proxies, and each of them can be set up using a single step or multiple steps.

To specify the type of proxy, use the --type or -t argument. A proxy can be either DIRECT, STATIC or PAC:

$ catchpoint proxy --type=STATIC   
OR   
$ catchpoint proxy -t STATIC

To set the proxy location, format depends on proxy type:
DIRECT proxy type does not require location option.
STATIC proxy type; supply --host or --port (Optional) for Proxy Location. For --host use hostname or IPv4 IP address or IPv6 IP address.

$ catchpoint proxy --host=your.proxy.url --port=1234
OR   
$ catchpoint proxy -h your.proxy.url -P 1234

PAC - Supply the URL where PAC file using --host

$ catchpoint proxy --host=http://pac_file_url/file.p   
OR   
$ catchpoint proxy -h http://pac_file_url/file.p

You may have an authenticated proxy setup. You can provide the credentials as well and these are all encrypted.

Proxy server Username when using a Static Proxy.

$ catchpoint proxy --usr={{username}}  
 OR   
$ catchpoint proxy -u {{username}}

Proxy server password when using a Static Proxy. The password is encrypted.

The password must follow -p without space. If no password is provided; you will be prompted for the password.

$ catchpoint proxy --password=\[password\]   
OR   
$ catchpoint proxy -p\[password\]

To reset the proxy, use the DIRECT option.

$ catchpoint proxy --type=DIRECT  
OR   
$ catchpoint proxy -t DIRECT

Some more examples using all parameters together:

$ catchpoint proxy --type=PAC --host=http://pac_file_url/file.pac \> catchpoint proxy -t PAC -h http://pac_file_url/file.pac      
$ catchpoint proxy --type=STATIC --host=your.proxy.url --port=1234 –-usr={{username}} --password=\[password\]      
$ catchpoint proxy -t STATIC -h your.proxy.url -P 1234 -u {{username}}
-p\[password\]

Disable TLS 1.0 & 1.1

There's a config file that we deploy for nginx at /etc/nginx/sites-enabled/default.

In that file, you add a new line after the ssl_certificate_key line as such: ssl_protocols TLSv1.2;.

By default all three are enabled if nothing is specified. The run sudo catchpoint restart.

To verify that it was properly disabled, run openssl s_client -connect localhost:443 -tls1 (or -tls1_1).

This will ensure TLS 1.0 and 1.1 is turned off.

Troubleshooting Feature not working

If you are not able to troubleshoot your Linux Node on the portal, make sure your firewall is open and SELinux is disabled. To open firewall; follow these steps:

$ sudo firewall-cmd --add-service=https --permanent   
$ sudo firewall-cmd --reload

To disable SELinux; follow these steps:

Check the status of SELinux

$ sudo getenforce   
Enforcing

Set SELinux to be Permissive

$ sudo setenforce 0   
$ sudo getenforce   
Permissive

The above steps will set SELinuxto Permissive only temporarily until the next time your system reboots. To make this change permanent:

$ vi /etc/selinux/config

Change the setting SELinux=Enforcing to SELinux=Permissive and reboot your machine.