Transfering control panel resources between hosts and partitions

Overview

The mf-transfer script automates the transfer of resources between servers and between partitions. It handles both copying the data and also updating the control panel resource to reflect the new location.

The command copies data from the source location to the destination location. If you are moving a resource from one partition to another on the same server, then the source and destination servers will be the same.

The command line script should be run as root from the destination server. Run mf-transfer to get up-to-date usage instructions.

A new high level comments on how it works:

  1. You will be prompted for your control panel username and password so the control panel items can be updated via the API.
  2. If the source server is remote, you will be prompted to add the destination root ssh public key to the appropriate non-privileged user’s authorized_keys file on the source machine (and it will be automatically removed when you are done).

Downtime

When moving resources between servers we can expect:

  1. Mailboxes: for approximately 15 - 30 seconds (the time it takes to update the control panel to reflect the new server), users will get an error if they are trying to download their email or access a mailbox.
  2. Databases: Usernames are switched to read-only just before the database is dumped and remain read-only until the database is imported. That may or may not cause visible errors depending on the application using the database is designed (Drupal and Wordpress seem to display ok but will throw errors for anyone who is logged in).
  3. Web sites: No downtime!! If someone is uploading a file during the transition, there is a small chance they will upload the file, the site will transition and the file won’t be there, but then the file will appear with the final sync.

MOSHES

When transferring resources between two servers using the new infrastructure, no preparation is required. However, for MOSHES it is important to set things up first.

When working on individual hosting orders, be sure to move web sites last. Mailboxes and databases must be moved before web sites. It is possible to move things on a hosting order by hosting order basis, but it’s generally easier to first move all databases on a MOSH, followed by all email boxes, followed by all web sites.

Mailboxes

  1. Start by reviewing all domains currently delivering to the mosh: mf-mx-test
  2. If necessary, re-point the domain name to use [abc].mx.mayfirst.org.
  • For all domains, even ones using external DNS services, use the mx-convert-domain-to-new-mx-servers script to automatically adjust the MX record to use the three May First MX domain names, randomly assigning weights to distribute the load:
    1. Login to red001
    2. Become the red user: su - red -s /bin/bash
    3. cd /usr/local/share/red/ui/sbin/
    4. ./convert-domain-to-new-mx-servers.php DOMAIN
  • For domains not using May First:
    1. Ask the member to update their domain name by following our online instructions.
  1. Check for any remaining domain names that are still using the MOSH you are working on as the MX server via the control panel database: SELECT dns_fqdn, item_status FROM red_item JOIN red_item_dns USING (item_id) WHERE item_status IN (‘active’, ‘disabled’) AND dns_type = ‘mx’ AND dns_server_name = ‘MOSH.mayfirst.org’;
  2. Either delete unused records or use the mx-convert-domain-to-mx-servers script.
  3. Begin transferring mailboxes.

Databases

We have to install and configure ProxySQL before we can move databases. Those steps are documented in transitioning moshes to use network databases.

Once a database is transitioned, it will no longer be accessible via the phpMyAdmin installed on the mosh. Instead it will be available via https://phpmyadmin.mayfirst.org/.

Web sites

Notify users! Unlike mailbox and databases which are mostly transparent to the user, users will no longer be able to access their sites via ssh/sftp after the change until the follow the new ssh/sftp directions.

You can copy the this one.

Web sites have a lot more potential problems. To try to find them, log in to the MOSH and run: mf-transfer-flight-check which may identify them.

Also, while in theory there is no downtime, when moving a web site from a MOSH to the new infrastructure, a lot of changes are automated, including updating paths in configuration files and scheduled jobs. These are hard to get right and frequenly will result in failures of either the web sites or the scheduled jobs. So, it’s important to test each web site before (in case it was already broken) and after.

If you have trouble connecting to the database from the weborigin server, try running the following as root on the weborigin server:

proxysql-load

Moving hosting order without a web configuration

If you need to move a hosting order on a MOSH that does not have a web configuration, you can do it by logging into the red server, becoming the red user, and:

cd /usr/local/share/red/ui/sbin
./red-migrate-non-web-hosting-order <hosting-order-id> <new-server>

Is everything really off?

After moving everything off of a server, here are a few SQL queries to run to determine that all records have been properly moved.

When moving off of a MOSH or a weborigin server.

Replace “MOSH” with the mosh/weborigin in question and IPADDRESS with the IP address in use by the mosh/weborigin (repeat with each IP address).

--- Check hosting orders.
SELECT * FROM red_hosting_order WHERE hosting_order_status != 'deleted' AND hosting_order_host = 'MOSH.mayfirst.org';
--- Check items
SELECT * FROM red_item WHERE item_status != 'deleted' AND item_host = 'MOSH.mayfirst.org';
--- Check DNS MX records and CNAME records.
SELECT red_item.item_id, dns_fqdn, dns_type, dns_server_name FROM red_item JOIN red_item_dns USING (item_id) WHERE item_status != 'deleted' AND dns_server_name = 'MOSH.mayfirst.org';
--- Check DNS A records.
SELECT red_item.item_id, dns_fqdn, dns_type, dns_server_name FROM red_item JOIN red_item_dns USING (item_id) WHERE item_status != 'deleted' AND dns_ip = 'IPADDRES';

If there are any hosting orders without a web configuration, then the mf-transfer script will not work.

But, you can still finish the job by logging into the control panel server and running: ./red-migrate-non-web-hosting-order <hosting_order_id> <weborigin>, e.g.: ./red-migrate-non-web-hosting-order 123 weborigin001.

When moving off of a network database server.

Replace “SERVER” with the database server in question.

SELECT * FROM red_item JOIN red_item_mysql_user where item_host = 'SERVER.mayfirst.org' and item_status != 'deleted';
SELECT * FROM red_item JOIN red_item_mysql_db where item_host = 'SERVER.mayfirst.org' and item_status != 'deleted';

Note: Sometimes a MySQL user is left behind. The only way to fix it is (replace NNN with the item_id):

UPDATE red_item SET item_status = 'soft-error', item_host = 'NEWSERVER.mayfirst.org' WHERE item_id = NNN;

Then, locate the item in the control panel, edit it, and submit.