IB Nodes Down

IB Nodes Down Alert

Alert ID: ib_nodes_down Category: Integration Broker Default threshold: Immediate

What This Alert Detects

This alert triggers when there are entries in the PeopleSoft table PSNODESDOWN. An entry in this table indicates that a message node is blocked or offline for one or more service operations.

When Integration Broker attempts to publish an asynchronous message to an external or remote node and encounters a connection failure or network hiccup, it inserts a row into PSNODESDOWN for that node and service operation (MSGNODENAME, TRXTYPE, IB_OPERATIONNAME, VERSIONNAME, EXTOPERATIONNAME, ROUTINGDEFNNAME). While that row exists, subsequent outbound publication contracts (PSAPMSGPUBCON) to the node stay in NEW or RETRY status and will not transmit.

Why Nodes Stay Stuck in PSNODESDOWN (The HTTP GET Ping Behavior)

When an outbound asynchronous integration enters retry status, Integration Broker begins polling the target endpoint using an HTTP GET request (with Transactionid: PING_NODE), even when the node or routing is configured to send HTTP POST requests. Integration Broker will not resume delivering queued messages until that HTTP GET ping returns an HTTP 200 status code.

If the external endpoint responds 200 OK to HTTP GET requests, the block clears automatically once the remote system recovers. However, many REST APIs and webhook receivers only accept POST on their target URL and return 404 Not Found or 405 Method Not Allowed to an HTTP GET. In that scenario, Integration Broker never receives an HTTP 200 response from its retry ping and the node remains stuck in PSNODESDOWN indefinitely.

Severity Logic

What Gets Checked

The alert queries PSNODESDOWN for any active rows. For each row found, it groups the results by message node and counts the number of blocked transactions.

Alert Details

Each alert item includes:

• The name of the blocked message node (MSGNODENAME)
• The count of blocked service operation routings recorded in PSNODESDOWN
• Contextual SQL (SELECT and DELETE statements scoped to MSGNODENAME)
• A link to the Node detail page in psLens showing the blocked transactions table and connection properties

Configuration

alerts:
  checks:
    ib_nodes_down:
      enabled: true

How to Respond

Verify Target Connectivity and Gateway Logs

1. Click the node link in psLens to review the node’s PRIMARYURL and connector properties on the node or routing definition.
2. Confirm that the external system is back online, DNS resolves from the gateway host, and TLS certificates in the Integration Gateway keystore have not expired.
3. Inspect the Integration Gateway error log on the web server for the underlying failure (SSLHandshakeException, ExternalSystemContactException, HTTP status errors, or SNI hostname issues requiring ig.UseDomainName.ExternalOperationNames in integrationGateway.properties):
   • PS_HOME/webserv/peoplesoft/applications/peoplesoft/PSIGW.war/WEB-INF/errorLog.html
   • PS_HOME/webserv/peoplesoft/applications/peoplesoft/PSIGW.war/WEB-INF/msgLog.html
   • Note: <ConnectorParameters> values in errorLog.html are URL-encoded; URL-decode them to inspect the exact Method and URL sent by the gateway.

Force a Retry from the Transaction Retry Queue

Once the target system is reachable, try forcing a retry from PIA first:

1. Navigate to PeopleTools > Integration Broker > Service Operations Monitor > Monitoring > Asynchronous Services.
2. Open the Publication Contracts tab.
3. Click the Transaction Retry Queue link.
4. Select the blocked node/operation and force a retry.

Clear PSNODESDOWN via SQL and Bounce the IB Domain

If forcing a retry from the Transaction Retry Queue does not clear the block (for example, because the target endpoint does not return HTTP 200 to an HTTP GET ping):

1. Inspect and delete the blocked row(s) from PSNODESDOWN (use the Contextual SQL at the top of this page when arriving from a psLens deep link, or replace <MSGNODENAME> below):

   SELECT * FROM PSNODESDOWN WHERE MSGNODENAME = '<MSGNODENAME>';
   
   DELETE FROM PSNODESDOWN WHERE MSGNODENAME = '<MSGNODENAME>';
   

2. Bounce (restart or cycle) the Integration Broker Pub/Sub application server domain (PeopleTools > Integration Broker > Service Operations Monitor > Administration > Domain Status) so the dispatcher resumes processing queued publication contracts.