Basic diagnostics

This article contains commands to identify causes of incorrect operation of the platform, as well as commands to restart the platform and some services to restore their work. To run commands, connect to the server with the platform via SSH.

General diagnostics

The section contains a list of commands for the first diagnostic step. These commands will eliminate basic problems and reduce troubleshooting time.

Operating system (OS) version

If the OS of the platform server is not supported, the installation will end with an error or the platform will not work properly. To determine the OS version, run the command:

cat /etc/*release

See BILLmanager 6 documentation for a list of supported operating systems:

• for BILLmanager Hosting&Cloud - in the Server Requirements article;
• for BILLmanager Advanced, Startup - in the Server Requirements article.

Server date and time

During periodic synchronization with the license server, the date and time are checked. If an incorrect date or time is set on the server with the platform, the platform will be blocked or its operation will be incorrect. To determine the current date and time on the server, run the command:

date -R

Disk space and RAM usage

For the platform to work correctly, the free disk space and RAM must meet the requirements specified in the BILLmanager documentation:

• for BILLmanager Hosting&Cloud - in the Server Requirements article;
• for BILLmanager Advanced, Startup - in the Server Requirements article.

Also, backups will not be created if there is not enough free space or RAM. To check the amount of used disk space and file system data, run the command:

df -hT

To check the information about RAM, run the command:

free -h

Inodes

Inodes is the structure in which file metadata is stored. The platform will not work correctly if the server is out of inodes, even if there is free disk space. Typical behavior when there is a shortage of inodes includes reduced performance, inability to create files, incorrect output of information in the platform interface. To check the number and proportion of inodes used in the filesystem, run the command:

df -i

Troubleshooting Linux systems

Examine the system logs to troubleshoot your Linux system. Below are tools for troubleshooting and searching for errors in the server logs.

Circular kernel buffer

One way to find out if the system is working incorrectly is to look at the kernel log using the dmesg utility. The kernel records all events in a circular buffer while the system is booting and running. dmesg  will allow you to examine the kernel messages and also identify hardware-related problems. To search for problems, run this command:

dmesg | grep -i -E 'error|failed|critical|bug|panic'

The journalctl utility

You can use the journalctl utility to analyze the logs and detect system problems. The utility displays the logs of the Linux system services. To detect abnormal behavior of a Linux system, run the command:

journalctl | grep -i -E 'error|failed|critical|bug|panic'

Licensing

Trial license

Trial license is intended for getting acquainted with the product and can be activated only once on one server. Also, you cannot activate the trial after the commercial version. To check if a BILLmanager license was activated on the server earlier, run the command:

cat /usr/local/mgr5/var/licctl.log | grep 'Got' 

Linking a license to a server

If the synchronization with the license server is lost, the platform will not start. To check the license information on the server, run the command:

/usr/local/mgr5/sbin/licctl info billmgr

If the output does not show the license number, it means that the license is not linked to a server. To activate the license on the server, run the command:

/usr/local/mgr5/sbin/licctl fetch billmgr

If you get the error Failed to get license when trying to activate the license:

1. Generate a new license key in your client area. Enter Products/Services → ISPsystem licenses → select a license → click Edit → click Generate new key.

2. Activate the license with the command:

   /usr/local/mgr5/sbin/licctl fetch billmgr KEY

   Comment

   KEY —  new key value

If generating a new key does not help, check the /usr/local/mgr5/var/licctl.log for diagnostics.

Payments

If a payment is successfully credited on the payment system side and its status has not been updated in BILLmanager, start manual synchronization with the payment system to update the payment status. To do this, run the command:

/usr/local/mgr5/sbin/mgrctl -m billmgr paymethod.checkpay

Restarting the platform

Restarting the platform can solve a number of basic problems, such as freezing, incorrect display of information in the interface, incorrect behavior of service processing modules. Also, restarting the platform is necessary to save the settings of payment modules, and service processing modules, when you change the logging level and other settings. To restart the platform, run the command:

/usr/local/mgr5/sbin/mgrctl -m billmgr -R

Logging

This section gives an example of diagnostics with the main log of the billmgr.log platform. Other logs may also be required for diagnostics. For example, in case of payment problems, information about errors will be recorded in the logs of the payment module, and in case of problems with reselling services — in the log of the interaction between the selling and reselling billing systems. The list of all logs is located at /usr/local/mgr5/var/. For the names of the logs, see the corresponding section in the BILLmanager 6 documentation. For example, logs to diagnose Paymaster payment problems are listed in the PayMaster article.

The main log of the platform billmgr.log records all the main events and errors. To check for errors in the main log, run the command:

grep 'ERROR' /usr/local/mgr5/var/billmgr.log

To check the logs at the moment, run the command:

tail -f /usr/local/mgr5/var/billmgr.log

Any problem can be diagnosed using the command to display the logs at the moment. To do this:

1. Open the section in the BILLmanager interface where the incorrect behavior occurs.
2. Connect to the server with the platform via SSH.
3. Open the real-time log with the command tail -f /usr/local/mgr5/var/billmgr.log.
4. Reproduce the incorrect behavior in the interface.

As a result of these actions, the log will display an error and the request that precedes it. This will help identify the cause of the incorrect behavior.

Threads and increased logging level

You can find out more information about the error by increasing the logging level and checking the log thread with the error.

To get additional information:

1. Increase the logging level for the log in question. To do this, add the line "name_log.* 9" to the file /usr/local/mgr5/etc/debug.conf.

   Comment

   name_log — the name of the log before the dot. For example, for billmgr.log you should enter billmgr

2. Restart the platform:

   /usr/local/mgr5/sbin/mgrctl -m billmgr -R

3. Identify the thread with the error. To do this, open the required log with a search by the error:

   grep 'ERROR' /usr/local/mgr5/var/name_log.log

   Comment

   name_log.log — the name of the log

   The output will show the error thread. Example of response:

   Mar  6 18:33:01 [483562:72787] libmgr ERROR Error: Type: 'missed' Object: 'function' Value: 'payment_cash_register.check'
   Mar  6 18:36:01 [483562:72797] libmgr ERROR Error: Type: 'missed' Object: 'function' Value: 'payment_cash_register.send'

   Comment

   483562:72787 —  log thread

4. Check the log file on the thread found:

   grep '483562:72787' /usr/local/mgr5/var/name.log.log

   This command will display detailed information about the error.

5. After diagnostics, reset the logging level to the default so that the detailed logs do not take up free space. To do this, delete or comment the line "name_log.* 9” added to the file /usr/local/mgr5/etc/debug.conf.

   Comment

   name_log — the name of the log before the dot

Searching for information in the database

The section contains basic database (DB) queries. To make other queries, use data about the BILLmanager database structure. For more information about the database structure, see the article Database structure in the BILLmanager 6 documentation.

To query the database, connect to it with a command:

mysql billmgr

Referral program

If a promo code is not activated, a query to the DB will help reveal information about the client's attachment to the referral program. To display information on the client's referral program, run a query:

Select * from account2project where account='ID_client'\G; 

The query will display information about:

• the partner program that the client uses from the provider;
• the partner from whose link the client was attracted;
• the referral program, through which the client is attracted.

Service processing modules

If an incorrect processing module is assigned to a service or no processing module at all is asigned, there will be problems with activating or deleting the service. Also, a "reset” of the processing module may occur when a service is transferred within a reselling billing service.

To check the service processing module, run the query:

select processingmodule from item where id =<item_id>;

License details

The query to collect license data will display all license parameters, including the name, remote ID and activation key. To display the license parameters, run the query:

select * from itemparam where item=<item_id>\G

The resulting output will display the data:

• intname - the name of the parameter. For example, lickey is the name of the parameter for the license key;
• value - the value of the parameter. For example, lickey will display the value of the key, and import_remote_id will display the value of the deleted identifier.

Provider

Query to the database will allow you to get the current status of the provider, linkage of a tariff plan to it and other parameters.

To check the active providers, run the query:

select id, active from project\G;

To find out which provider the tariff plan is linked to, run the query:

select project from pricelist where id in (select pricelist from item where id='<item_id>');

Filters

With the help of queries to the database, you can apply filters that are not in the interface of the platform. This section contains queries to display data on customers with an active Promised Payment and a non-zero credit limit.

To display a list of services for which a promised payment was activated, run the query:

SELECT * FROM promisepayment;

To display the list of clients with an active credit limit, run the query:

select account from subaccount where creditlimit>0.0000