---
title: "Spark UI"
description: "How to configure links from Corridor job details to the Spark Web UI and Spark History Server."
---

Apache Spark can be setup in multiple ways in cluster mode like `yarn`, `mesos` and also in standalone mode.
Apache Spark provides a suite of web user interfaces (UIs) that the user can use to monitor the status and resource
consumption of the Spark cluster.

Corridor uses Spark for running jobs through the platform and through integrated notebooks.
Corridor provides a direct link from Corridor job in the platform to the underlying Spark UI.

The user has the ability to see Spark execution related information for currently running jobs and for
previously run jobs, using Spark Web UI and Spark History Server UI, by configuring the options outlined below.

:::note
Spark History Server has to be setup as a prerequisite to see Spark execution of previously run jobs.
(Refer: [Setting up Spark History Server](https://sparkbyexamples.com/spark/spark-history-server-to-monitor-applications/))
:::

## Configurations

The user would need to add the appropriate configurations in the [`api_config.py`](/technology/installation/configuration-setup/api/) file where the API process is running. For clusters deployed on Kubernetes, see [Connecting to a Spark Kubernetes Cluster](/technology/customization/spark-k8-cluster/).

#### SPARK_RM_UI_ENABLED

This setting controls whether to show SparkRM-UI link for the job details, with two possible values:

- True: This will take the values from the settings `SPARK_UI_URL` and `SPARK_HISTORY_SERVER_URL` and show the
  RM UI link in the Job Details
- False: This will ignore the values of `SPARK_UI_URL` and `SPARK_HISTORY_SERVER_URL` in the configs and not show
  the RM UI link in the Job Details

Example:

```python
SPARK_RM_UI_ENABLED = True
```

#### SPARK_UI_URL

This setting points to the URL of the Spark UI for currently running Spark jobs

Requires: `SPARK_RM_UI_ENABLED = True`

Possible values:

- `SPARK_UI_URL = <web_url>` (`<web_url>` should be of the type `http(s)://<host>:<port>/../jobs/`)
- `SPARK_UI_URL = 'from-spark'` (supported only for **standalone Spark** installations )

Example:

```python
SPARK_UI_URL = 'http(s)://<cluster_url>:<port>/cluster/app/'
# OR
SPARK_UI_URL = 'from-spark'
```

:::note
`'from-spark'` should only be set for standalone Spark installations, correct behavior is not guaranteed for Spark setup in cluster mode (`yarn`, `mesos`, etc.)
:::

#### SPARK_HISTORY_SERVER_URL

This setting points to the URL of the Spark UI for previously run Spark jobs

Requires: `SPARK_RM_UI_ENABLED = True` **and** `SPARK_UI_URL = <web_url>`

Possible values:
`SPARK_HISTORY_SERVER_URL = <web_url>` (`<web_url>` should be of the type `http(s)://<host>:<port>/../history/`)

:::note
This setting can only be configured if Spark is run in cluster mode (`yarn`, `mesos`, etc.)
:::

Example:

```python
SPARK_HISTORY_SERVER_URL = 'http(s)://<history_server_url>:<history_server_port>/history/'
```