Skip to main content

Data Processing Overview

Data processing steps form the foundation of Jetty workflows, enabling file operations, data transformations, storage integration, and external service connections. These steps handle everything from simple file reading to complex data analysis pipelines with trajectory-based storage management.

Category Structure​

Tools Steps (6 steps)​

File operations, utilities, and external integrations:

  • text_concatenate - Combines multiple text files into a single output
  • read_text_file - Reads text content from storage backend
  • download_image - Downloads images from URLs with metadata extraction
  • save_text_file - Saves text content to storage backend
  • add_image_metadata - Extracts and adds EXIF metadata to images
  • webhook_notify - Sends HTTP notifications to external services

Trajectory Storage Context​

All file operations integrate with trajectory storage:

  • Storage Providers: S3, GCS, local filesystem
  • Automatic Path Resolution: Trajectory-specific paths
  • Content Type Handling: Automatic MIME type detection

Path Patterns​

  • Relative Paths: outputs/file.txt → trajectory-scoped
  • Absolute Paths: /shared/data/file.txt → shared across workflows
  • Dynamic Paths: outputs/{{step_name}}_{{timestamp}}.txt

Next Steps​

Getting Help​

  • Review individual step documentation for detailed parameters
  • Check the Flow Library for data processing examples
  • Consult trajectory documentation for path resolution patterns