MPI Operation Manual

Starting/Stopping the MPI

Use the mpictl script to start or stop an MPI node.

Run the script with the -h argument to learn about the available options and commands:

$ $MPI_HOME/bin/mpictl.sh -h

Logging

Refer to the Log4j Configuration documentation.

Credentials

The MPI application uses TLS/SSL client and root certificates stored in Java Key Store (JKS) format.
Keystore files, once created and populated, need to be referenced for usage in the MPI Configuration file.

For creating and manipulating keystores, Java keytool is used. The keytool is part of the Java Development Kit (JDK).

The keytool expects certificates to be imported in either DER format or PEM format (Base64 encoded DER format).

Client certificates

MPI distinguishes 2 types of client certificates:

  • ClientCertificate on level Merchant (XPath in mpi-configuration.xml: /Merchant/MerchantAcquirer/ClientCertificate):

    Points to a keystore containing a certificate and a private key which are used during the Verify Enrollment phase, when the MPI is communicating to the Directory Server on behalf of the merchant. It is used by the Directory Server to authenticate the merchant. If the merchant is authenticated by username and password, this field may be left out empty. ClientCertificate on level Merchant takes precedence over ClientCertificate on Acquirer level (see below), when resolving which keystore to use.

  • ClientCertificate on level Acquirer (XPath in mpi-configuration.xml: /Acquirer/Bin/ClientCertificate):

    Merchants are associated with acquirers. Merchants may authenticate to the Directory Server using the ClientCertificate defined on level Acquirer. If there is no ClientCertificate defined on Merchant level, the Acquirer ClientCertificate is used.

Keystore and key pair

To generate the keystore and the key pair execute the following command:

keytool -genkey -alias client -keyalg RSA -keysize 2048 -keystore client.jks -dname "CN={COMMON NAME},O={ORGANZIATION},L={LOCATION},ST={CITY},C={COUNTRY}"

Replace the placeholders in the distinguished name (e.g. COMMON NAME) with the appropriate values for the Merchant or Acquirer.

Change the name of the keystore and the alias as you wish. For this and the following examples client.jks and client are used.

Important: The passwords for the keystore and the key have to be the same in order for the MPI to be able to access the keystore and the key.

Certificate Signing Request (CSR)

The Certificate Signing Request (CSR) can be created with the following command:

keytool -certreq -alias client -file client.csr -keystore client.jks 

NOTE: Visa's MPI Client Certificate Guide requires the signing algorithm to be SHA1WithRSA. Use the sigalg option to specify the algorithm.

This command creates the client.csr file from the previously created keystore.

Obtaining a signed certificate

The CSR (generated in the previous step) needs to be sent to the acquirer for the given scheme in order to receive a valid signed certificate.

Certificate Import

Once the client.crt file is obtained from the Certificate Authority, it can be imported into the existing keystore.

First import the CA certificate as a trusted entry:

keytool -import -trustcacerts -alias ca -file ca.crt -keystore client.jks

Then use the following command to import the client certificate:

keytool -import -trustcacerts -alias client -file client.crt -keystore client.jks

NOTE: It is important that the certificate is imported under the same alias as the existing alias with the private key. (i.e. in this case the alias is "client")

Existing Private Key and Certificate Import

For migrating existing client private keys and certificates, export them to PKCS#12 format and then use the Java keytool to import them in JKS format.

Use openssl to create the PKCS#12 file (src.p12) from your client certificate (client.crt), the private key (client.key) and the CA certificate (ca.crt):

$ openssl pkcs12 -export -out src.p12 -inkey client.key -in client.crt -certfile ca.crt

Then import the PKCS#12 file into a Java keystore:

$ keytool -importkeystore -srckeystore src.p12 -srcstoretype PKCS12 -destkeystore client.jks

Note: use the same password when exporting the PKCS#12 file and when creating the destination JKS keystore.

Client Certificate Renewal

When renewing a client certificate, the existing CSR can be sent to the CA in order to issue a new valid certificate for the same private key.

After the CA responds, the new certificate can be imported into the client keystore. In order not to overwrite any existing valid certificates, first list the available aliases in the key store:

keytool -list -keystore client.jks 

If the initial CSR is reused, the client private key already exists in the key store and must be cloned under a new alias:

keytool -keyclone -alias client -dest client-new -keystore client.jks

Note: private key password and keystore password must be the same.

Import the renewed client certificate (client-new.csr) under the newly created alias:

keytool -import -trustcacerts -alias client-new -file client-new.crt -keystore client.jks

Note that the MPI sets the Java Security property ssl.KeyManagerFactory.algorithm to NewSunX509 in order to properly support multiple certificates in the same Keystore. If multiple certificates are available, it prefers valid to expired ones.

Union Pay signature keys

The private key for creating signature values in UnionPay messages and the certificate for signature validation are stored in keystores on either Merchant or Acquirer level. The same configuration structure and keystore storage is used as for 3DS client credentials.

These credentials are not used for SSL communication.

Root certificates

MPI distinguishes 2 types of root certificates:

  • TrustedRootCertificate (XPath in mpi-configuration.xml: /Scheme/DirectoryServer/TrustedRootCertificate):

    Points to a keystore containing a root and all intermediate certificates which are used during the Verify Enrollment phase to verify the SSL server certificate presented by the Directory Server.

  • SignatureValidationRootCertificate (XPath in mpi-configuration.xml: /Scheme/SignatureValidationRootCertificate for 3DS and /UpopParameters/SignatureValidationRootCertificate for UPOP):

    Points to a keystore containing a certificate which is used during the Payer Authentication phase, when the MPI verifies the validity of the digitally signed XML received from the ACS or UPOP.

Certificate import

Once the root.crt file is received, it can be imported into a keystore. Use the following command to import the root certificate:

keytool -import -trustcacerts -alias root -file root.crt -keystore trusted-root.jks

On the prompt, when asked if this certificate should be trusted, enter: yes.

Verify that the certificate and the alias have been added to the keystore:

keytool -list -keystore trusted-root.jks -alias root
Root certificate renewal

If the TrustedRootCertificate or the SignatureValidationRootCertificate are nearing the expiration date, a new certificate may be added to the respective keystore.

Given a keystore file named "trusted-root.jks", first list all aliases in the keystore:

keytool -list -keystore trusted-root.jks 

Given new certificate file named "new-certificate.crt", existing alias name "root" and new alias "root-new", importing the certificate is done with the following command:

keytool -import -trustcacerts -alias root-new -file new-certificate.crt -keystore trusted-root.jks

On the prompt, when asked if this certificate should be trusted, enter: yes.

Verify that the certificate and the alias have been added to the keystore:

keytool -list -keystore trusted-root.jks -alias root-new

Note that the MPI sets the Java Security property ssl.KeyManagerFactory.algorithm to NewSunX509 in order to properly support multiple certificates in the same Keystore. If multiple certificates are available, it prefers valid to expired ones.

Root certificate changed

If the root certificate (TrustedRootCertificate or SignatureValidationRootCertificate) are no longer valid, they should be removed from the keystore and a new valid certificate should be imported.

Given a keystore file named "trusted-root.jks", first list all aliases in the keystore:

keytool -list -keystore trusted-root.jks 

Given new certificate file named "new-certificate.crt" and existing alias name "root" first we remove the old invalid certificate.

keytool -delete -alias root -keystore root.jks

Then we import the new valid certificate:

keytool -import -trustcacerts -alias root -file new-certificate.crt -keystore trusted-root.jks

On the prompt if this certificate should be trusted, enter: yes.

Verify that the certificate and the alias have been added to the keystore:

keytool -list -keystore trusted-root.jks -alias root-new

MPI configuration

Entity configuration (Merchants, Acquirers and Schemes)

Refer to the MPI Configuration documentation for details about the configuration format.

Configuration properties

Refer to the MPI Configuration Properties documentation for details about the configuration options.

Retry mechanism for Verify Enrollment request to the Directory Server

Directory servers can be configured with multiple directory server endpoints, so that in case of a failure in one - MPI can try contacting the next available endpoint. MPI will use the retry mechanism for all DS hierarchy of exceptions, unless it is at the last endpoint. The exceptions for which the retry mechanism will trigger are the following:

  • DS_HTTP_MESSAGE_CONVERSION_ERROR
  • DS_HTTP_STATUS_CODE_ERROR
  • DS_TIMEOUT_ERROR
  • DS_REST_CLIENT_ERROR

Reloading the MPI Configuration at runtime

The MPI supports reloading the MPI Configuration without restarting the MPI.

NOTE: The MPI exposes an endpoint http://<host>:<port>/mpi/admin/config/reload which can be used to reload the configuration using an HTTP POST request. An additional dryRun=true request parameter can be sent to the endpoint to have the MPI only validate the MPI configuration without actually reloading it.

For convenience, the mpictl.sh script can be used with the following command to validate the MPI configuration:

$ $MPI_HOME/bin/mpictl.sh -n 1 configtest

After the new configuration has passed the validation, the MPI Configuration can be reloaded with the following command:

$ $MPI_HOME/bin/mpictl.sh -n 1 configreload

NOTE: The script will parse the server-$NODE_ID.xml found in the $MPI_CONFIG_HOME folder to resolve the node host and port. It will then send an HTTP POST request (using curl) to the config-reloading-endpoint.

NOTE: The configtest and configreload commands apply to only one MPI node. The -n option is used to specify the MPI node that should reload the configuration. In a setup with more than one MPI node, the reloading of the MPI Configuration must be performed on all nodes. Before issuing the configreload command, make sure that all MPI nodes use or have the same MPI Configuration.

Reloading the MPI License at runtime

The MPI supports reloading the MPI License wihtout restarting the MPI.

NOTE: The MPI exposes an endpoint http://<host>:<port>/mpi/admin/license/reload which can be used to reload the license using an HTTP POST request. An additional dryRun=true request parameter can be sent to the endpoint to have the MPI only validate the MPI license without actually reloading it.

For convenience, the mpictl.sh script can be used with the following command to validate the MPI license:

$ $MPI_HOME/bin/mpictl.sh -n 1 licensetest

After the new license has passed the dryRun reloading, the MPI License can be reloaded with the following command:

$ $MPI_HOME/bin/mpictl.sh -n 1 licensereload

NOTE: The script will parse the server-$NODE_ID.xml found in the $MPI_CONFIG_HOME folder to resolve the node host and port. It will then send an HTTP POST request (using curl) to the config-reloading-endpoint.

NOTE: The licensetest and licensereload commands apply to only one MPI node. The -n option is used to specify the MPI node that should reload the configuration. In a setup with more than one MPI node, the reloading of the MPI License must be performed on all nodes.

Security

Access permissions

You should protect the MPI Configuration file using Unix file permissions as it contains keystore passwords.

Make sure the user running the MPI application is still able to read the file.

Password obfuscation

Passwords (keystore passwords and merchant acquirer passwords) in the MPI Configuration can be obfuscated.

Create an obfuscated password by using the mpitool script as follows:

$ $MPI_HOME/bin/mpitool.sh obfuscate my_password

To make the MPI application aware of using obfuscated passwords, you have to set the corresponding MPI Configuration Property:

mpiConfigXml.useObfuscatedPasswords=true

MPI exposed HTTP endpoints

MPI exposes the following endpoints to which access can be restricted:

Admin endpoints
Admin utility endpoints
  • http(s)://<host>:<port>/mpi/admin/util/merchants
    • exposes the configured merchants in JSON format
  • http(s)://<host>:<port>/mpi/admin/util/currencies
    • exposes the configured currencies in JSON format
  • http(s)://<host>:<port>/mpi/admin/util/countries
    • exposes the configured countries in JSON format
  • http(s)://<host>:<port>/mpi/admin/util/schemes
    • exposes the configured schemes in JSON format
3-D Secure endpoints
Union Pay endpoints:

Cluster setup

Multiple MPI nodes can be run in a cluster for:

  • High availability
  • Scale-out deployments
  • Zero-downtime deployments

When starting multiple MPI nodes on the same machine, use the -n command line argument with the mpictl script and specify the node ID.

You have to setup a load balancer to distribute the workload to the MPI nodes in the cluster.

If the MPI is configured to maintain an internal session for the requests in a 3-D Secure transaction, either setup sticky sessions (to ensure the same MPI node receives all requests belonging to the same 3-D Secure transaction) or setup session replication by adjusting your Ehcache Configuration. Alternatively you can use an external session and pass the session data to the MPI via the MPI API.

Session handling strategies

The MPI needs to maintain data between the createPaReqIfEnrolled and validatePaRes calls or the 3DS transactions, and between the createAuthenticationReq and validateAuthenticationRes for Union Pay transactions. You should choose the session handling strategy that best suits your setup. Sticky sessions and session replication can be used in a cluster setup where the MPI internally maintains a session, while the external session can be used both in a cluster setup and a single node MPI installation.

Sticky sessions

Configure your load balancer to use sticky sessions and make sure to send additional Cookies or Header values as needed by your load balancer when sending requests to the MPI.

Before shutting down an MPI node (e.g. for upgrading), make sure that there are no more running transactions (e.g. by instructing your load balancer to not send any new transactions to this node and wait till all running transactions are completed).

Session replication

As an alternative to using sticky sessions, you can use the MPI's support for session replication.

For details about session replication configuration with the MPI, refer to the Ehcache Configuration documentation.

When using session replication, every MPI node in the cluster may process any request of a 3-D Secure or Union Pay transaction. I.e. every node can resume sessions started on another node.

External session

If you want to manage the session on your own, you can use an external session and pass the additional arguments (i.e. the session data) needed by the MPI to validate the PaRes and MsgRes via the MPI API.

This session handling strategy can be used in a single MPI node installation, as well as in a cluster setup.

If the MPI is configured to use external session, the SessionData element is expected to be present in the PaResValidationRequest.

SessionData for the 3DS API contains the following values:

  • paReqCreationTime - needed for validation of the timeout constraint
  • pan - needed for validation of the pan in PARes
  • acsUrl - the URL of the ACS from where the response was received, so that MPI can notify the ACS in case of an error
  • ThreeDSecurePAReq - needed for validation of values in PARes

SessionData for the Union Pay API contains the following values:

  • transactionTime - needed for validation of the timeout constraint
  • needed for validation of values of MsgRes
    • accessType
    • merchantId
    • orderId
    • acqInsCode
    • transactionCurrency
    • transactionAmount
    • accountNumber

Errors and troubleshooting

Exceptions are logged to the MPI application log (see Log4j Configuration for details) on level ERROR.

For every exception during request processing, an error code is logged that can be used to categorize the error.

Refer to the Error Codes documentation for a description of the error codes used by the MPI.

For most error codes the merchant can not intervene. In frequent cases of timeouts when communicating with the Directory Server or while waiting for a response from the ACS, increasing the corresponding timeout should be considered (see also MPI Configuration).

Reporting

Obtain info from the log files

Search the log files using grep command (refer to Log4j Configuration documentation for location of the log files).

  • Find error by error code
    grep DS-REST mpi.2013-04-03.log
  • Find error by error description
    grep "Error while trying to communicate with the Directory Server." mpi.2013-04-03.log
  • Get all log messages connected to transaction id or session id
    grep x72RheEAN7TX1Aw5NBQg mpi*.log

    or

    grep f26f7244-e273-4bce-a0b8-d2cf096f7136 mpi*.log 
  • Navigate to some exception in mpi.log
    less mpi.2013-04-03.log

    and search for error code (e.g /DS-REST)

  • Navigate to some date
    less mpi.2013-04-03.log

    and search for date (e.g /2013-04-03 14:39:51,436)

Statistics reports

Extract the performance statistics from the log files by using the mpictl script showstats command.

$ $MPI_HOME/bin/mpictl.sh -n 1 showstats

Monitoring

Dashboard

The MPI provides an HTML dashboard reachable under /mpi/admin/dashboard.

The dashboard shows information about:

  • Performance and failure statistics
  • Errors
  • Configuration
  • License

Note that performance and exceptions statistics will be lost when MPI application is restarted.

Exceptions Dashboard

When exception occurred in MPI in the last day, alert icon is shown on the dashboard page.

By opening the Exceptions tab, you can check what type of exception occurred.

By clicking on an error row with errors (rows highlighted in red), one can see the details about the last exceptions.

Configuration Dashboard

By opening the Configuration tab, one can observe the current MPI configuration. MPI will notify of any expired or soon to expire certificates with a warning icon displaying the cumulative number of problematic certificates from all the entities (Merchants, Acquirers, Directory Servers, Schemes).

By opening the Configuration tab menu, one can observe the configuration entities (Merchants, Acquirers, Directory Servers, Schemes) with a number of their own problematic certificates.

When selecting an item of the Configuration tab menu, you can see the entity specific view.

Merchants
Acquirers
Directory Servers
Schemes
License Dashboard

When MPI license is no longer valid or is soon to expire, alert or warning icon is shown on the dashboard page.

By opening the License tab, you can check the license.
Alert icon will be shown when either MPI version does not match or the license valid date has passed.
Warning icon will be shown if license is soon to expire.

JMX

The MPI exposes various MBeans to facilitate monitoring via JMX.

You can connect to the MPI e.g. by using jconsole which comes bundled with the Java Development Kit (JDK) distribution.

Performance and failure statistics are exposed under org.perf4j:type=StatisticsExposingMBean,name=Perf4J. Check the Perf4j Tags for an overview of the available statistics tags corresponding to operations in the MPI. The statistics information is aggregated over the configured time slice (see Log4j Configuration for details).

For every operation, the following success and failure information is available:

  • Count - Number of operation executions
  • Min - Minimum time used for the operation in ms
  • Max - Maximum time used for the operation in ms
  • Mean - Mean time used for the operation in ms
  • StdDev - Standard deviation
  • TPS - Transactions per seconds (i.e. operation executions per second)

Directory Server Simulation Mode

For testing purposes MPI can be configured to be run with Directory Server simulation, in which case the real Verify Enrollment request to the Directory Server will be replaced with fixed response outcomes provided in a properties file.

In order to set the Directory Server simulation mode, enable the dsResponseSimulation configuration property, and provide a property file containing the Directory Server response outcomes.
For the format and the location of the simulation property file check the configuration of dsResponseSimulation.resourceLocation in MPI Configuration Properties documentation.