Create a Cloud Connector to Connect to your Cloud Storage

Owned by Vytas Glambinskas
Nov 12, 2023
3 min read

Integrating your data pipelines via platform connectors allows you to add data to the platform and to take data off the platform, to use within your local environments.

It is often a good idea to setup your cloud connectors as soon as possible as sometimes it requires assistance from other departments in your company.

Note: Use the Snowflake web interface (Snowpipe) or any other supported Snowflake ETL software to integrate data from or to your cloud storage connector.

A direct connection to Snowflake is coming soon.

Note: You must have an Technician role to perform this action

Your connectors within your cloud accounts must have been created, else setup a cloud-agnostic SFTP connector

To create an connector:

  1. Find and click the connectors button under the Manage menu in the main navigation bar.

  2. Click Create connector.

  3. Enter the connector Name.

  4. Enter an connector Description.

  5. Select the Type of connector you want to set up.
    Follow the instructions in the appropriate section below.

S3

  1. Choose Type > S3.

  2. Insert your S3 path.
    Example: s3://my-bucket/my-data
    This is the bucket name you set when you created your.

  3. Select your AWS Region from the list.

  4. Insert your IAM credentials (if requested)

    1. Insert your Amazon access key

    2. Insert your Amazon secret key

  5. Click Create.

  6. Follow the Your AWS S3 bucket policy steps

    • Navigate to the bucket in the AWS console S3 browser

    • Enter the `Permissions` tab and `Bucket Policy` option

    • Copy and paste the generated policy statement below into the text editor

    • Save

  7. Click Close

Still not sure? Watch this video on how to set up an AWS S3 bucket.

 

 

GCS

  1. Choose Type > GCS.

  2. Enter your bucket name and optional subdirectory (path/to/dir). The Platform uses these locations when publishing and exporting to the connector.

  3. Enter the GCS bucket name.

  4. (Optional) Enter the GCS subdirectory.

  5. Select GCP Key File (if requested)

  6. Click Create.
    The New GCS connector information page appears.
    Help: This page informs you that the connector is ready and that the member account and roles are created. It also provides guidance on how to apply the required roles and access permissions to the bucket member account. Ensure bucket access control is set to Fine-grained.

  7. Go back to Google Console and log in

  8. Go to your bucket

  9. Click on Roles > Add

  10. Add New principals (This is the member email given to you on platform once you create your connector)

  11. Add the following roles:

    1. Storage Legacy Bucket Writer

    2. Storage Legacy Object Owner

12. Click Close.

 

 

Azure Blob Storage

  1. Choose Type > Azure Blob Storage.

  2. In the Shared Access Signature section:

    1. Enter the Blob SAS URL.

    2. (Optional) Insert a Blob prefix.

  3. Click Create.
    The new connector ready message box appears.

  4. Use the access permissions instructions to add the displayed member details to your permissions tab.

  5. Click Close.

SFTP

(available to some customers only, contact Support if you would like to know more)

  1. Choose Type > SFTP

  2. Click + Create key to create an SSH key. You can create up to ten SSH keys. SSH keys can be downloaded as .PEM files and used to access the SFTP server.

Related Pages

https://harbrgroup.atlassian.net/wiki/spaces/HDV5/pages/2772993185

Create an S3 Bucket

Create an Azure Blob SAS token and URL

https://harbrgroup.atlassian.net/wiki/pages/createpage.action?spaceKey=HDV5&title=Source%20your%20Data%20Product%20from%20an%20Endpoint

https://harbrgroup.atlassian.net/wiki/spaces/HDV5/pages/2772993687