Setting Up a Globus File Transfer: MAX IV to DTU

Globus is a platform designed to simplify and streamline data management and transfer in research and academic environments. It offers researchers an efficient way to move data between different endpoints such as computers, servers, and storage systems.

In this tutorial, we'll guide you through the process of setting up a Globus file transfer between MAX IV and DTU Compute.

Requirements

Before starting with the tutorial, let's ensure that you have everything needed.

MAX IV Checklist

• Make sure you can log in to your MAX IV user account using DUO authentication.
• Ensure you have access to the MAX IV white network via VPN and that OTP (One-Time Password) is operational. You can test this here.

If you encounter any issues during these processes, refer to the Max IV user guides.

DTU Checklist

• You'll need a working DTU username. Test the operational status of your username by logging into the Qim platform here.
• Ensure you have a project space to save your files. Typically, projects are located here.

Step 1: Link accounts on Globus

The first step needed is to link your MAX IV and DTU accounts at Globus

For this, go to app.globus.org

You'll be redirected to the DTU login page, where you should sign in using your DTU credentials.

After login, your home page at Globus should look similar to this:

On the File manager tab, click Search and look for a Collection called MAX IV Laboratory

More than one result may be found, but the official one we're looking for contains the following icon:

When Max IV is selected, an Authentication/Consent Required message should appear:

Click continue and the following should appear:

Click on MAX IV Laboratory Globus Endpoint (maxiv.lu.se) and you should be redirected to the Max IV SSO page:

Select MAX IV Account and you'll be redirected to the login page, proceed with username, password and OTP:

After successful login, you'll be redirected back to Globus and asked to log back in with your primary identity. Simply click Continue

There will be an agreement message, click Allow to proceed:

There might be a message asking for reauthentication, if that's the case just proceed with the login again:

There's a possibility that you encounter the following message: Your credentials do not provide sufficient access to this endpoint

Completely logging out from Globus and logging in again using your DTU credentials is likely to solve the issue.

In case the error persists, it could be that you don't have sufficient permissions at MaxIV. If that's the case, contact MAX IV.

When successfully connecting to the MAX IV endpoint, you should be able to explore the files:

Step 2: Configure your endpoint at DTU

Now that the MAX IV account is linked, the next step is to start a personal Globus endpoint at DTU, so that Globus can perform a file transfer between the two servers.

First, open your terminal and connect to the DTU transfer server:

ssh <username>@transfer.gbar.dtu.dk

Once you're connected, download the latest Globus personal:

wget https://downloads.globus.org/globus-connect-personal/linux/stable/globusconnectpersonal-latest.tgz

Then, extract the contents by running:

tar -xvf globusconnectpersonal-latest.tgz

Go to the directory where Globus was extracted:

cd globusconnectpersonal-3.2.5/ ( be aware that your version might be different)

and start the setup process: globusconnectpersonal -setup

Open the provided address in your browser. In the page that loads, give a name to your personal endpoint that you'll remember later.

Click Allow and you'll receive a Native App Authorization Code.

Copy this code, and paste it back into the terminal. Write your end-point name and then you should get the message setup completed successfully

Verify that this process was done correctly by going back to https://app.globus.org/ and on the right pane:

1. Click on the search bar
2. Click on Your Collections

The endpoint you created should be visible there:

Step 3: Start the Globus Personal endpoint

Now that the Globus endpoint is configured, we need to start the server.

As we registered the endpoint, it is findable at Globus, but the message Globus Connect Personal Is Not Running will appear in case you try to explore the files. So we need to start the server and keep it running in the background.

To start the globus connect personal server, use the following command:

nohup globusconnectpersonal -start -restrict-paths '/dtu/3d-imaging-center/projects/' > globus.log 2>&1 &

To verify if the server is running, you can use the command ps aux | grep globus

Step 4: Data transfer

On the left pane select the MAX IV Laboratory endpoint, and on the right pane select the DTU endpoint you created.

Navigate to the desired directory, and select the files (or folders) that you want to transfer. Click the Start ▶ button and in a few seconds there should be a notification saying that the transfer request was submitted.

You can easily verify the transfer status on the Activity tab. The browser can now be closed, and the transfer will continue in the background.

You will receive a summary email from no-reply@globus.org when the transfer is complete.