How to import data from BILLmanager 4

Preliminary steps

Before you migrate to BILLmanager 5, please read carefully the list of changes. 

Migration from BILLmanager 4 to BILLmanager 5 consists of two steps:

1. Conversion of the database into a new format
2. Conversion and import of BILLmanager 4 files to a new server 

Before you migrate to a production server, we strongly recommend that you perform a test import and check that information has been migrated to BILLmanager 5 successfully. Complete the following steps to prepare a server:

• Enable all the supported features in BILLmanager 4. The migration rules may refer to the tables presented in the database only with certain features enabled.
• Install a required BILLmanager version. You can import data only on a freshly installed system. We do not recommend that you create users, clients, and other objects.
• Suspend cron. Alternatively, you can disable all cron tasks associated with BILLmanager. Сron tasks look like this:

/usr/local/mgr5/sbin/billmaintain ...
/usr/local/mgr5/sbin/mgrctl -m billmgr ...
/usr/local/mgr5/notify/ntinternal ...
/usr/local/mgr5/notify/ntemail ...
/usr/local/mgr5/notify/ntsms ...

• Create the file /usr/local/mgr5/etc/billmgr.DoNothing. It prevents any billing actions with any external systems. 

touch /usr/local/mgr5/etc/billmgr.DoNothing

Follow the above recommendations to get a fully-functional BILLmanager that will NOT perform any operations with services in your control panels, send emails, or charge clients.

After you have prepared the testing environment, you can upload the data to the MySQL (MariaDB) database.

Converting the database 

There are two ways to import the database: directly from the server where BILLmanager 4 is installed or from a database dump on your local machine.

The first way:

• • You need to enable remote access for MySQL users (root by default)
  • It can slow down BILLmanager 4
  • Data transfer through a network may slow down the data conversion process
  • If the connection disrupts, you will have to start the operation from the beginning
  • In the database, new objects might be created with reference errors to the objects that have not been received yet

The second way:

• • BILLmanager 4 gets blocked during the creation of a new dump
  • Migration can be repeated a few times, each new time with less time wasted

Once you have prepared the database, execute the following command to convert the data:

/usr/local/mgr5/sbin/billimport --command check --billing billmgr4 --db <db name> [--host <db host> [--user <db user> [--password <db password>]]]

Now you can connect the database and check current conversion rules.

If the checking was successful, you can start the data import:

/usr/local/mgr5/sbin/billimport --command import --billing billmgr4 --db <db name> [--host <db host> [--user <db user> [--password <db password>]]]

where: 

• --command — the action that the billimport utility shall perform
• --billing — billing system name that you are importing your data from (when you import data from BILLmanager 4 to BILLmanager 5, this value equals to billmgr4, if you import data from BILLmanager 4 Advanced to BILLmanager 5 Corporate, it equals to billmgr4adv)
• --db — the name of the database from where the data for the conversion is selected
• --host — the server to retrieve the data from; this is the optional parameter. The default value is localhost
• --user — the username to connect to the database; this is the optional parameter. The default value is root.
• --password — the user password; this is the optional parameter. By default, the password is not used.

Additional parameters:

• limit — the value size (the number of records) that will be selected from the database you want to import. The import process includes several steps, so the MySQL (MariaDB) limits on database packages wouldn't exceed. The default value is 10000 records.
• lhost — the server that the data conversion script connects to. You shall specify this parameter if you want to upload data to a database that is already used by BILLmanager 5.
• luser — the username to connect to the BILLmanager 5 database. You shall specify this parameter if it differs from the name in the BILLmanager 5 configuration file.
• lpassword — the user password to connect to the BILLmanager 5 database.
• ldb — the database the data will be imported to, if its name differs from the current database.

During migration, billimport selects the data from the source database, converts it into a required format, and saves in the BILLmanager 5 database. Please make sure to form the database structure and add data before you start. I.e. the database shall look like you have just installed BILLmanager 5 on a fresh server.

During data conversion, billimport launches some BILLmanager 5 functions that are executed on the data that you use. It means, if you are making the import to another database, you shall call the following functions manually:

• fix.docnumber — converts settings for document numbers for companies from BILLmanager 4 format to BILLmanager 5 format.
• fix.currency.status — adds the active currency to a certain provider.
• fix.orderperiod — enables/disables order periods used in the product type settings.
• fix.addon.compound — converts add-ons with subtypes and add-ons with the "Enumeration with quantity" type into add-ons with the "Client can choose" type.
• fix.processingmodule.param — encrypts data to connect to processing modules.
• fix.addon.manual — converts add-ons manually added to services into a new format.
• fix.certificate.cn — transfers SSL certificate domain name to service parameters.
• fix.country.profile.param — creates parameters of the imported payers; 
• fix.itemip — re-assigns IP addresses according to the new IP address system.
• fix.billorder.item — converts orders from users that have not been processed yet into the cart format.
• fix.invoice.realamount — converts the internal structure of invoices into a new format.
• fix.subscription — converts information about client notification subscription into a new format.
• fix.domain.dopparam — converts domain parameters into a new format.
• fix.account.country — specifies a country for clients based on the payer data.
• fix.accountgroup_condition — converts rules for automatic assigning to groups into a new format.
• fix.department.rights — converts access permissions to functions into a new format and grants full permissions.
• fix.fraudgateway — converts anti-fraud gateway parameters.
• fix.gateway — converts mail gateway and mail receiving parameters.
• fix.globalindex.rebuild — starts data indexing for quick search.
• fix.import.period — checks service order period and automatic renewal period and changes it to a suitable period if needed.
• fix.itemtype.intname — converts the internal names of product types to exclude similar names;
• fix.measure — converts units of measure used in BILLmanager 4 into BILLmanager 5 units. Removes duplicate values.
• fix.modules — installs missing processing modules, mail gateways, and payment methods.
• fix.pricelist.domain.intname — sets a correlation between domain registration tariff plans and top-level domains.
• fix.profile.localization — corrects payers and companies data storage method and adds the documents localization parameter.
• fix.pmauto — adds the system handler pmauto, if needed;
• fix.support.incident — converts information about support tickets into a new format.
• fix.taskdepartment — selects a responsible department for imported tasks.
• fix.tax.expense — if taxes are added to the price specified in the tariff plan, it recalculates the unpaid expenses and open orders.
• fix.remove.internal.itemtype — deletes the system product type and tariff plans; 
• fix.update_expiredate — starts synchronization of license update periods.
• fix.itemparamip — imports IP addresses for virtual machines and dedicated servers to the "IP addresses" section.
• processing.getconfig — retrieves information from processing modules configured.

After the data has been imported and the functions have been executed, you can proceed to the additional steps for importing data.

Special aspects of data import

Provider tax policy:

• If tax rates were not added to the price and were set up for at least one company, taxes will be added to the service price.
• If taxes were added to the service price, tax rates will be detached from the companies.

Add-ons that use subtypes and add-ons with the "Enumeration with quantity" type are converted into "Client can choose" add-ons.

Service and IP-address information added to history will be marked as "History synchronization".

If periods that were created in BILLmanager 4 are missing in BILLmanager 5, the system will choose the most similar period according to it's length.

Data that cannot be converted

• Corporate version — information about partners, their clients, and other associated objects;
• Advertisement data which was added to Email attachments;
• Tariff plan agreements;
• Automatic invoices configuration;
• Information on inquiries;
• Remote document exchange. Contractor information shall be requested manually;
• Knowledgebase;
• Notifications that were not sent;
• Running service operations;
• Support center statistics;
• Business hours.

Notes

Client IDs, their personal accounts, and service codes remain unchanged during the import process.

Financial and billing information, information on services and tariff plans will be imported.

Notification and documents templates, branding settings, images in templates and other additional files which were not included in the structure of BILLmanager 4 won't be imported.

Support tickets from the Support center section are divided into notifications (newsletters), tickets and tasks with description.

If you convert data from BILLmanager 4 to a different version of BILLmanager 5, some data may get lost, since the BILLmanager 5 database doesn't have all the fields and tables that are used in BILLmanager 4.

After finishing your database import, conversion of user access rights can be incomplete because it runs in the background. To make sure that this process is over, please execute the following command:

ps aux | grep billfix

Importing and converting files 

When you migrate from BILLmanager 4 to BILLmanager 5, besides database information importing, you shall import tickets attachments and user configuration files. It also shall be converted into a new format.

Importing file attachments to tickets:

• use scp or rsync to copy the /usr/local/ispmgr/var/mailattach directory from the BILLmanager 4 server to the BILLmanager 5 server and rename it to /usr/local/mgr5/var/ticket_attach.
• once copying is done, run the function to convert the attachments into the new BILLmanager 5 format.

/usr/local/mgr5/sbin/mgrctl -m billmgr fix.ticket_message_fileattach

Importing and converting user configuration files:

• Use scp or rsync to copy the /usr/local/ispmgr/var/userconf directory from the BILLmanager 4 server to the BILLmanager 5 server and rename it to /usr/local/mgr5/tmp_userconf.
• Once copying is done, run the function:

/usr/local/mgr5/sbin/mgrctl -m billmgr fix.userconf

Importing user avatars:

Use scp or rsync to copy the /usr/local/ispmgr/var/userconf directory from the BILLmanager 4 server into the /usr/local/mgr5/skins/userdata/avatar_files directory to BILLmanager 5 server.

Converting service and IP address history

Service and IP address history could be converted and imported to BILLmanager 5. Due to significant differences in storage methods, conversion can be done by executing the special BILLmanager 5 function.

The function call:

/usr/local/mgr5/sbin/mgrctl -m billmgr fix.history host=$host user=$user password=$password db=$db

The command parameters:

• $host — IP address of the server where BILLmanager 4 database is located
• $user — database username
• $password — database user password
• $db — database name

If service processing modules were not deleted, you can run conversion at any time after data importing.