Skip to main content

JupyterLab on Edge

Jupyter notebooks are the most popular environment for exploratory analysis and interactive programming. Edge makes them even more powerful for you. To get started, click on the "Analysis" tile on your Scientific Workbench. Once you are inside the Analysis App, you can work with your Jupyter notebooks just like any other Jupyter or JupyterLab system.

/img/edge_notebook.png

You can launch a "Get Started!" notebook and see a quick demonstration of what you can do in notebooks on Edge:

/img/edge_notebook_get_started.png

Connection to Files on Edge​

On the left sidebar of your Analysis App, you will notice that there are two file browsers. One is the built-in local JupyterLab filesystem where you can upload and read data for analysis. This is also where you create and save your notebooks. But there is also a second browser specific to JupyterLab on Edge that points to Files on Edge. This is the shared file space of the Edge organization where you have launched your Analysis App from.

You can move data between your local JupyterLab filesystem and Files on Edge with a simple drag and drop. This works in both directions. If you want to create a local copy of a shared file to analyze and modify it in a notebook; or if you want to share the results of your work in JupyterLab with your team, all you have to do is to drag and drop files between the two file browsers:

/img/edge_drag_and_drop.png

Jupyter notebooks on Edge​

On Edge, you also get a predefined object in each notebook, called edge. This object serves as your programmatic access point to the rest of your Edge organization. At any time, you can evaluate edge (run a cell with the edge object by itself) to get an overview of the Edge object and its functionality:

>>> edge
This object serves as your entry point to access data and files in Edge.

    You can browse files stored in the Edge internal file
    system by using the ".files" attribute.  Here's an example:

    [1]: edge.files.list()
    ['myfile.jpg', 'myfile.csv', 'myfolder']

    ...
>>> type(edge)
edge.edge_session.EdgeSession

The edge object connects you to the rest of the Edge ecosystem. For example, here's how you can list uploaded files and folders in Edge's internal storage:

>>> edge.files.list()
['Example Folder',
 'Experiments.txt',
 'Imported Data',
 'My Data',
 'Projects',
 'data.csv',
 'data_cleaned.csv',
 'image001.png',
 'image002.png',
 'image003.png']

From your edge object, you can also access data from remote data sources:

>>> edge.sources.list()
['data-warehouse-sql', 'petstore-api', 'data-lake-s3']

>>> lake = edge.sources.get('data-lake-s3')
>>> lake.root.list()
['Experiment data', 'SEM micrographs']

>>> lake.root.open('Experiment data').list()
['01f23764ad0b4642a374f68349c5504c.png',
 '0a311d9ac6cc4fdb9aace7d927bae892.bin',
 '232d50ebf0714315953aa2b0edd0c9f6.bin',
 '367581c15fc24175af8fbd78cd13f0fd.png',
 '5c622f05c7ec44e1b3749961c2ae56e4.png',
 '770a34f466ad4c76bbde849337be48b6.bin',
 '92a62cccfcb849b885c29de64114c3ae.bin',
 '92da96436830471195a12f56bb4af6c8.png',
 'd8d73e96bff44632985ad563eb4397cd.bin',
 'dc852dfb0b89442ab918297db9b8d727.png',
 'e004927530c74ed7ac8c510744645d55.png',
 'e38326ab94494f50ba1050b3ee704eee.bin']

Installing Python packages​

A large number of Python packages useful for scientific data analysis are preinstalled. You can also add additional packages using edm (the Enthought Deployment Manager) or pip (the default Python package manager) in a JupyterLab terminal:

$ edm install <package-name>
$ pip install <package-name>
caution

Packages can't be installed into a running Python environment. Before installing packages via edm or pip, be sure to shut down any active kernels. To do this, go to the Kernels menu and select Shut Down All Kernels. Please note that you will lose any in-memory values for your notebooks.

We recommend using the Enthought Deployment Manager (edm) unless the package you want isn't available from Enthought's package repositories. Packages in edm have been extensively tested to make sure they work together. You can read more about EDM in the official EDM docs.

From a notebook tab, you can return to the launcher page using the large blue "+" button at top left (screenshot attached below). From there you can launch new notebooks, terminal windows and more.

/img/edge_analysis_launcher.png

Connecting to Github​

If you have a repository hosted on a version control service such as GitHub or GitLab, you can clone it into your workspace by using the built-in git extension.

Clicking on "Git" in the top menu brings up the option to "Clone a Repository", which will then prompt you to enter the GitHub or GitLab repository URI. Be sure it starts with "https://".

Screenshot of the dialogue that opens when you clone a repo. It asks for you to enter the URL.

tip

On GitHub, you can find your repository URI by opening the page for the repository and selecting the blue "< > Code" button. In GitLab, open the repository and look for the blue "Clone" button. You can copy the URI from the box that opens.

caution

If your repository is not publicly accessible, you will see a prompt asking for your credentials to the version control service. For security, it is recommended that you enter a personal access token instead of your password. You can find information on creating a personal access token on the GitHub or GitLab documentation pages.

Dialogue saying to &quot;enter credentials for remote repository&quot;

Resetting Edge​

If you get into trouble, shut down any running notebooks and run the reset-edge command from the terminal. This will restore your package set to the "factory default" settings:

$ reset-edge

Resetting Edge will:

* remove all custom Python packages and environments
* restore your Python packages to factory default

You will NOT lose any files or notebooks.

Continue and reset Edge (y/n)?

When you see the prompt above, enter y to confirm the reset. Your packages will be restored to their default settings.

Automatic Shutdown​

To preserve compute resources, any running kernel or terminal that is inactive for 30 minutes will be terminated. After that, if the Analysis App detects no activity for 4 hours, it will automatically shut down.

Avoiding Automatic Shutdown​

There are times when you want to extend the life of the Analysis App beyond the set automatic shutdown time. To this end there is an edge-keepalive script that allows you to indicate the Analysis environment is busy even if you are not actively using it:

$ edge-keepalive <duration>

Where duration is an integer followed by a time scale, minutes ('m'), hours ('h'), or days ('d').

$ edge-keepalive 2h

Keeping alive for 2 hours:
Keep alive remaining time:  2:00:00
Keep alive remaining time:  1:59:00
...

Jupyter AI on Edge​

Jupyter AI, an extension for Jupyter notebooks, seamlessly integrates generative AI capabilities into the Analysis App. This enhancement facilitates increased productivity and provides the ability to interact with large language models (LLMs) directly within the Jupyter environment. Key offerings of Jupyter AI include:

  • Support for a wide range of generative model providers and models.
  • A native chat UI in JupyterLab that enables you to work with generative AI as a conversational assistant.
  • An %%ai magic that turns the Jupyter notebook into a reproducible generative AI playground.

Model Providers​

Jupyter AI supports a wide range of generative model providers and models. To use Jupyter AI with a specific provider, it is necessary to install the corresponding Python packages and configure the API key within your environment or the chat interface.

For utilizing models from the model providers supported, an authentication token is required; you can set it using:

# NOTE: Replace 'PROVIDER_API_KEY' with the credential key's name,
# and replace 'YOUR_API_KEY_HERE' with the key.
%env PROVIDER_API_KEY=YOUR_API_KEY_HERE
info

OpenAI is supported by default but other model providers can also be supported upon request.

For more specific instructions for each model provider, refer to the model providers documentation.

The chat interface​

Jupyter AI includes a chat interface that puts you in conversation with Jupyternaut, a native chat UI in JupyterLab that presents the idea of interacting with an AI model conversationally. Jupyternaut communicates primarily through text, and it can also interact with files in JupyterLab. It can answer questions as a general-purpose AI assistant, include selections from your notebooks with your questions, insert AI-generated output directly into your notebooks, learn from and ask questions about your local files, and generate notebooks from a prompt. Jupyternaut respects your privacy, only accessing information when you explicitly send chat commands, ensuring that it reads your data only when needed.

To open the chat interface, navigate to the left side panel of your Analysis App, click on the "chat" icon, and then click the "Start Here" button.

/img/jupyterai_chat_ui.png

This will prompt you to select both a language model and an embedding model. The language model engages with users in the chat panel, generating responses based on prompts. Meanwhile, the embedding model comes into play when learning from and querying local data. Before diving into the chat interface, make sure to provide your API keys for the model providers you selected.

/img/jupyterai_settings.png

Once you have set all the necessary keys, click the “back” (left arrow) button positioned in the upper-left corner of the Jupyter AI side panel. The chat interface now appears, and you can simply use the message box at the bottom to start engaging.

/img/jupyternaut.png

tip

You are able to use the following commands:

  • /learn : Teach Jupyternaut about files on your system
  • /ask : Ask a question about your learned data
  • /generate : Generate a Jupyter notebook from a text prompt
  • /clear : Clear the chat window
  • /help : Display this help message

For more information on Jupyternaut, refer to the chat interface documentation.

The %ai and %%ai magic commands​

Jupyter AI extends its functionality with magic commands that are executable both in notebook and the command-line interface. You can use Jupyter AI with the %%ai magic command.

tip

To explore available options and commands, execute %ai help, which provides insights into both %ai line magic and %%ai cell magic commands.

Before you send your first prompt to an AI model, ensure you load the extension by running the following code in a notebook cell or shell:

%load_ext jupyter_ai_magics

This command won't generate any output. Once the extension is loaded, and the environment variable is configured, you can run %%ai cell magic commands and %ai line magic commands.

The %%ai cell magic allows you to engage with a language model of your choosing by utilizing a specified prompt. A distinct global model ID distinguishes each model, formed as a string with the format <provider-id>:<local-model-id>. Here, <provider-id> signifies the unique identifier of the provider, while <local-model-id> represents the identifier of the model constrained within that provider. The prompt is initiated on the second line of the cell.

For example, if you want to send a text prompt to the anthropic provider using the model ID claude-v1.2, input the following code into a cell and execute it:

%%ai anthropic:claude-v1.2
A function that computes the lowest common multiples of two integers

An example using OpenAI's chatgpt can be seen below as well:

/img/jupyterai_magics.png

For more information on %%ai magic commands, refer to the magic commands documentation.