Using an Amazon S3 Connection

Once you’ve successfully connected your Amazon S3 bucket to Nexadata, you can begin using the connection to bring data into Nexadata (via Datasets) or send data back to S3 (via Pipeline Outputs). This article will guide you through both processes.

Here are the key capabilities:

Import data from S3 by selecting your bucket and specifying a file path.
Export transformed data to S3 using either:
- Dynamic output (appends a timestamp to each file), or
- Static output (overwrites the same file).
Query available S3 buckets and browse files within your connection.
Customize output paths and formats as needed for your workflows.

Let’s walk through each setup process step-by-step.

Part 1: Creating a Dataset from S3

Use this process to ingest data stored in S3 into Nexadata for use in Pipelines.

Step 1: Create a New Dataset

Navigate to the Datasets section in Nexadata.
Click Create Dataset.
Under Data Connection, select your configured S3 connection (e.g., Nexadata Demo S3).

Step 2: Configure Dataset Properties

Name: Provide a unique name for your dataset.
Data Connection: Ensure your Amazon S3 connection is selected.

Step 3: Select Data Format

Choose the format of your file stored in S3:

Tabular (CSV or TXT)
JSON (coming soon)
Parquet (coming soon)

Step 4: Specify S3 Bucket and File Path

Start typing to search available buckets (e.g., nexadata-demo).
In S3 Path, provide the full file name (e.g., SIMPLE_FACT.txt).

Step 5: Set Delimiter (for Tabular)

Choose your delimiter:

Comma (,)
Tab
Semicolon (;)
Pipe (|)

Step 6: Submit the Dataset

Review the setup and click Submit.
The dataset is now ready to use in any Nexadata pipeline.

Part 2: Configuring an S3 Output in a Pipeline

Once your Nexadata pipeline is complete, use S3 as an output destination to store your transformed results.

Step 1: Add an Output to the Pipeline

Navigate to the Customized Outputs section under the Pipeline Setup tab.
Click Add Output and select your configured Amazon S3 connection.

Step 2: Choose Output File Type

Select how the file will be written to S3:

Dynamic (recommended): Adds a timestamp to each file name.
Static: Replaces the file at the specified S3 path on each run.

📌 Dynamic is ideal for versioned outputs. Static is best for overwriting the same reference file.

Step 3: Provide S3 Details

S3 Bucket: Enter your S3 bucket name (e.g., nexadata-demo).
S3 Path: Provide the full path with filename (e.g., exports/report.csv).

Step 4: Save the Output

Click Save to confirm and register the output configuration with your pipeline.

Tips and Best Practices

Dynamic Outputs help preserve historical pipeline results by creating timestamped files.
Use Static Outputs only when you’re sure overwriting is acceptable.
Always ensure proper IAM permissions to list and write to S3 buckets.
Organize S3 paths by project or date for easier tracking.
Double-check file formats and delimiters when importing tabular data.