# IB Nodes Down

> This alert triggers when there are entries in the PeopleSoft table PSNODESDOWN.

---

Site [llms.txt](/llms.txt)

---

<div class="alert alert-warning border shadow-sm mb-4 psl-prospect-cta" role="note">
  <div class="d-flex flex-column flex-lg-row align-items-lg-center justify-content-between gap-3">
    <div>
      <strong>New to psLens?</strong> This page documents one specific alert. To see how it appears on the dashboard,
      what operators investigate, and how teams tune it, start with a live walkthrough.
    </div>
    <div class="d-flex flex-wrap gap-2">
      <a class="btn btn-sm btn-primary" href="/contact/">Book a Demo</a>
      <a class="btn btn-sm btn-outline-primary" href="/docs/alerts/">Browse Alerts</a>
    </div>
  </div>
</div><div id="pslens-context-panel" class="card border-info mb-4 d-none">
  <div class="card-header bg-light text-info py-2 fw-bold d-flex align-items-center border-bottom border-info-subtle">
    <i class="bi bi-info-circle-fill me-2"></i>
    <span>Tailored Operational Context</span>
  </div>
  <div class="card-body p-0">
    <ul class="list-group list-group-flush">
      <li id="row-db" class="list-group-item d-flex align-items-center justify-content-between py-2 d-none">
        <strong>Target Database:</strong>
        <span id="ctx-db" class="badge bg-secondary font-monospace">&mdash;</span>
      </li>
      <li id="row-type" class="list-group-item d-flex align-items-center justify-content-between py-2 d-none">
        <strong>Context Type:</strong>
        <span id="ctx-type" class="badge bg-light text-dark border font-monospace text-uppercase">&mdash;</span>
      </li>
      <li id="row-severity" class="list-group-item d-flex align-items-center justify-content-between py-2 d-none">
        <strong>Alert Severity:</strong>
        <span id="ctx-severity" class="badge">&mdash;</span>
      </li>
      <li id="row-time" class="list-group-item d-flex align-items-center justify-content-between py-2 d-none">
        <strong>Triggered Time:</strong>
        <span id="ctx-time" class="text-muted small">&mdash;</span>
      </li>
      <li id="row-details" class="list-group-item py-2 d-none">
        <strong id="label-details" class="d-block mb-1">Firing Context:</strong>
        <code id="ctx-details" class="d-block p-2 bg-light border rounded small"
          style="white-space: pre-wrap; word-break: break-all;">&mdash;</code>
      </li>
      <li id="row-sql" class="list-group-item py-2 d-none">
        <div class="d-flex justify-content-between align-items-center mb-1">
          <strong id="label-sql">Contextual SQL:</strong>
          <button id="btn-copy-sql" type="button" class="btn btn-sm btn-outline-secondary py-0 px-2 small">
            <i class="bi bi-clipboard me-1"></i>Copy SQL
          </button>
        </div>
        <pre class="mb-0 p-2 bg-light border rounded small"><code id="ctx-sql" style="white-space: pre-wrap; word-break: break-word;">&mdash;</code></pre>
      </li>
    </ul>
  </div>
</div>

<script>
  (function () {
    const params = new URLSearchParams(window.location.search);
    const metadata = params.get('metadata');
    if (!metadata) return;

    try {
      
      const base64 = metadata.replace(/-/g, '+').replace(/_/g, '/');
      const jsonStr = decodeURIComponent(escape(window.atob(base64)));
      const data = JSON.parse(jsonStr);

      if (data) {
        let hasData = false;

        if (data.db) {
          document.getElementById('ctx-db').textContent = data.db;
          document.getElementById('row-db').classList.remove('d-none');
          hasData = true;
        }

        if (data.type) {
          document.getElementById('ctx-type').textContent = data.type;
          document.getElementById('row-type').classList.remove('d-none');
          hasData = true;
        }

        if (data.severity) {
          const severityBadge = document.getElementById('ctx-severity');
          const severity = data.severity.toLowerCase();
          severityBadge.textContent = severity.toUpperCase();
          if (severity === 'critical') {
            severityBadge.className = 'badge bg-danger';
          } else if (severity === 'warning') {
            severityBadge.className = 'badge bg-warning text-dark';
          } else {
            severityBadge.className = 'badge bg-info';
          }
          document.getElementById('row-severity').classList.remove('d-none');
          hasData = true;
        }

        if (data.t) {
          const date = new Date(data.t * 1000);
          document.getElementById('ctx-time').textContent = date.toLocaleString();
          document.getElementById('row-time').classList.remove('d-none');
          hasData = true;
        }

        if (data.details) {
          document.getElementById('ctx-details').textContent = data.details;

          
          const labelDetails = document.getElementById('label-details');
          if (data.type === 'object') {
            labelDetails.textContent = 'Object Metadata Details:';
          } else if (data.type === 'report') {
            labelDetails.textContent = 'Report Description:';
          } else {
            labelDetails.textContent = 'Firing Context:';
          }

          document.getElementById('row-details').classList.remove('d-none');
          hasData = true;
        }

        if (data.sql) {
          document.getElementById('ctx-sql').textContent = data.sql;
          document.getElementById('row-sql').classList.remove('d-none');
          const copyBtn = document.getElementById('btn-copy-sql');
          if (copyBtn && navigator.clipboard) {
            copyBtn.addEventListener('click', function () {
              navigator.clipboard.writeText(data.sql).then(function () {
                copyBtn.innerHTML = '<i class="bi bi-check2 me-1"></i>Copied';
                setTimeout(function () {
                  copyBtn.innerHTML = '<i class="bi bi-clipboard me-1"></i>Copy SQL';
                }, 2000);
              });
            });
          }
          hasData = true;
        }

        if (hasData) {
          document.getElementById('pslens-context-panel').classList.remove('d-none');
        }
      }
    } catch (e) {
      console.error('Failed to parse operational context metadata:', e);
    }
  })();
</script>

## IB Nodes Down Alert

**Alert ID:** `ib_nodes_down`
**Category:** Integration Broker
**Default threshold:** Immediate

### What This Alert Detects

This alert triggers when there are entries in the PeopleSoft table `PSNODESDOWN`. An entry in this table indicates that a message node is blocked or offline for one or more service operations.

When Integration Broker attempts to publish an asynchronous message to an external or remote node and encounters a connection failure or network hiccup, it inserts a row into `PSNODESDOWN` for that node and service operation (`MSGNODENAME`, `TRXTYPE`, `IB_OPERATIONNAME`, `VERSIONNAME`, `EXTOPERATIONNAME`, `ROUTINGDEFNNAME`). While that row exists, subsequent outbound publication contracts (`PSAPMSGPUBCON`) to the node stay in `NEW` or `RETRY` status and will not transmit.

### Why Nodes Stay Stuck in `PSNODESDOWN` (The HTTP GET Ping Behavior)

When an outbound asynchronous integration enters retry status, Integration Broker begins polling the target endpoint using an **HTTP GET** request (with `Transactionid: PING_NODE`), even when the node or routing is configured to send `HTTP POST` requests. Integration Broker will not resume delivering queued messages until that HTTP GET ping returns an **HTTP 200** status code.

If the external endpoint responds `200 OK` to HTTP GET requests, the block clears automatically once the remote system recovers. However, many REST APIs and webhook receivers only accept `POST` on their target URL and return `404 Not Found` or `405 Method Not Allowed` to an HTTP GET. In that scenario, Integration Broker never receives an HTTP 200 response from its retry ping and the node remains stuck in `PSNODESDOWN` indefinitely.

### Severity Logic

|                 Condition                  | Severity |
| ------------------------------------------ | -------- |
| Message node entry exists in `PSNODESDOWN` | Critical |

### What Gets Checked

The alert queries `PSNODESDOWN` for any active rows. For each row found, it groups the results by message node and counts the number of blocked transactions.

### Alert Details

Each alert item includes:

- The name of the blocked message node (`MSGNODENAME`)
- The count of blocked service operation routings recorded in `PSNODESDOWN`
- Contextual SQL (`SELECT` and `DELETE` statements scoped to `MSGNODENAME`)
- A link to the Node detail page in psLens showing the blocked transactions table and connection properties

### Configuration

```yaml
alerts:
  checks:
    ib_nodes_down:
      enabled: true
```

|  Setting  | Default |          Description          |
| --------- | ------- | ----------------------------- |
| `enabled` | `true`  | Whether this check is active. |

### How to Respond

#### Verify Target Connectivity and Gateway Logs

1. Click the node link in psLens to review the node's `PRIMARYURL` and connector properties on the node or routing definition.
2. Confirm that the external system is back online, DNS resolves from the gateway host, and TLS certificates in the Integration Gateway keystore have not expired.
3. Inspect the Integration Gateway error log on the web server for the underlying failure (`SSLHandshakeException`, `ExternalSystemContactException`, HTTP status errors, or SNI hostname issues requiring `ig.UseDomainName.ExternalOperationNames` in `integrationGateway.properties`):
   - `PS_HOME/webserv/peoplesoft/applications/peoplesoft/PSIGW.war/WEB-INF/errorLog.html`
   - `PS_HOME/webserv/peoplesoft/applications/peoplesoft/PSIGW.war/WEB-INF/msgLog.html`
   - Note: `<ConnectorParameters>` values in `errorLog.html` are URL-encoded; URL-decode them to inspect the exact `Method` and `URL` sent by the gateway.

#### Force a Retry from the Transaction Retry Queue

Once the target system is reachable, try forcing a retry from PIA first:

1. Navigate to **PeopleTools > Integration Broker > Service Operations Monitor > Monitoring > Asynchronous Services**.
2. Open the **Publication Contracts** tab.
3. Click the **Transaction Retry Queue** link.
4. Select the blocked node/operation and force a retry.

#### Clear `PSNODESDOWN` via SQL and Bounce the IB Domain

If forcing a retry from the **Transaction Retry Queue** does not clear the block (for example, because the target endpoint does not return HTTP 200 to an HTTP GET ping):

1. Inspect and delete the blocked row(s) from `PSNODESDOWN` (use the **Contextual SQL** at the top of this page when arriving from a psLens deep link, or replace `<MSGNODENAME>` below):

   ```sql
   SELECT * FROM PSNODESDOWN WHERE MSGNODENAME = '<MSGNODENAME>';

   DELETE FROM PSNODESDOWN WHERE MSGNODENAME = '<MSGNODENAME>';
   ```

2. Bounce (restart or cycle) the Integration Broker Pub/Sub application server domain (**PeopleTools > Integration Broker > Service Operations Monitor > Administration > Domain Status**) so the dispatcher resumes processing queued publication contracts.