Integrating your data pipelines via platform connectors allows you to add data to the platform and to take data off the platform, to use within your local environments.

It is often a good idea to setup your cloud connectors as soon as possible as sometimes it requires assistance from other departments in your company.

To create an connector:

1. Find and click the connectors button under the Manage menu in the main navigation bar.

2. Click Create connector.

3. Enter the connector Name.

4. Enter an connector Description.

5. Select the Type of connector you want to set up.
   Follow the instructions in the appropriate section below.

S3

1. Choose Type > S3.

2. Insert your S3 path.
   Example: s3://my-bucket/my-data
   This is the bucket name you set when you created your.

3. Select your AWS Region from the list.

4. Insert your IAM credentials (if requested)

   1. Insert your Amazon access key

   2. Insert your Amazon secret key

5. Click Create.

6. Follow the Your AWS S3 bucket policy steps

   • Navigate to the bucket in the AWS console S3 browser

   • Enter the `Permissions` tab and `Bucket Policy` option

   • Copy and paste the generated policy statement below into the text editor

   • Save

7. Click Close

Open

 

 

GCS

1. Choose Type > GCS.

2. Enter your bucket name and optional subdirectory (path/to/dir). The Platform uses these locations when publishing and exporting to the connector.

3. Enter the GCS bucket name.

4. (Optional) Enter the GCS subdirectory.

5. Select GCP Key File (if requested)

6. Click Create.
   The New GCS connector information page appears.
   Help: This page informs you that the connector is ready and that the member account and roles are created. It also provides guidance on how to apply the required roles and access permissions to the bucket member account. Ensure bucket access control is set to Fine-grained.

7. Go back to Google Console and log in

8. Go to your bucket

9. Click on Roles > Add

10. Add New principals (This is the member email given to you on platform once you create your connector)

11. Add the following roles:

    1. Storage Legacy Bucket Writer

    2. Storage Legacy Object Owner

12. Click Close.

Open

 

 

Azure Blob Storage

1. Choose Type > Azure Blob Storage.

2. In the Shared Access Signature section:

   1. Enter the Blob SAS URL.

   2. (Optional) Insert a Blob prefix.

3. Click Create.
   The new connector ready message box appears.

4. Use the access permissions instructions to add the displayed member details to your permissions tab.

5. Click Close.

SFTP

(available to some customers only, contact Support if you would like to know more)

1. Choose Type > SFTP

2. Click + Create key to create an SSH key. You can create up to ten SSH keys. SSH keys can be downloaded as .PEM files and used to access the SFTP server.

Related Pages

https://harbrgroup.atlassian.net/wiki/spaces/HDV5/pages/2772993185

Create an S3 Bucket

Create an Azure Blob SAS token and URL

https://harbrgroup.atlassian.net/wiki/pages/createpage.action?spaceKey=HDV5&title=Source%20your%20Data%20Product%20from%20an%20Endpoint

https://harbrgroup.atlassian.net/wiki/spaces/HDV5/pages/2772993687