job.proto
path mgmt/v1alpha1/job.proto
package mgmt.v1alpha1
Messages
ActivityFailure
Name | Type | Description |
---|---|---|
message | string |
ActivityOptions
Config that contains various timeouts that are configured in the underlying temporal workflow(s) and activities
Name | Type | Description |
---|---|---|
schedule_to_close_timeout | optional int64 | Total time that a workflow is willing to wait for an activity to complete, including retries. Measured in seconds |
start_to_close_timeout | optional int64 | Max time of a single Temporal Activity execution attempt. This timeout should be as short as the longest psosible execution of any activity (e.g. table sync). Important to know that this is per retry attempt. Defaults to the schedule to close timeout if not provided. Measured in seconds |
retry_policy | RetryPolicy | Optionally define a retry policy for the activity If max attempts is not set, the activity will retry indefinitely until the start to close timeout lapses |
AiGenerateSourceOptions
Name | Type | Description |
---|---|---|
ai_connection_id | string | |
schemas | repeated AiGenerateSourceSchemaOption | |
fk_source_connection_id | optional string | |
model_name | string | The name of the model to use |
user_prompt | optional string | Optionally provide a user prompt to give more context to the schema |
AiGenerateSourceSchemaOption
Name | Type | Description |
---|---|---|
schema | string | |
tables | repeated AiGenerateSourceTableOption |
AiGenerateSourceTableOption
Name | Type | Description |
---|---|---|
table | string | |
row_count | int64 |
AwsS3DestinationConnectionOptions
Name | Type | Description |
---|
AwsS3SourceConnectionOptions
Name | Type | Description |
---|---|---|
connection_id | string |
CancelJobRunRequest
Name | Type | Description |
---|---|---|
job_run_id | string | |
account_id | string |
CancelJobRunResponse
Name | Type | Description |
---|
ColumnError
Name | Type | Description |
---|---|---|
schema | string | |
table | string | |
column | string | |
errors | repeated string |
CreateJobDestination
Name | Type | Description |
---|---|---|
connection_id | string | |
options | JobDestinationOptions |
CreateJobDestinationConnectionsRequest
Name | Type | Description |
---|---|---|
job_id | string | |
destinations | repeated CreateJobDestination |
CreateJobDestinationConnectionsResponse
Name | Type | Description |
---|---|---|
job | Job |
CreateJobRequest
Name | Type | Description |
---|---|---|
account_id | string | The unique account identifier that this job will be associated with |
job_name | string | The unique, friendly name of the job. This is unique per account |
cron_schedule | optional string | Optionally provide a cron schedule. Goes into effect if the job status is set to enabled |
mappings | repeated JobMapping | |
source | JobSource | |
destinations | repeated CreateJobDestination | |
initiate_job_run | bool | Initially trigger a run of this job regardless of its status or cron schedule |
workflow_options | WorkflowOptions | Specify timeouts and other workflow options for the underlying temporal workflow |
sync_options | ActivityOptions | Specify timeout and retry options for data synchronization activities Data sync activities are any piece of work that involves actually synchronizing data from a source to a destination For the data sync and generate jobs, this will be applied per table |
virtual_foreign_keys | repeated VirtualForeignConstraint |
CreateJobResponse
Name | Type | Description |
---|---|---|
job | Job |
CreateJobRunRequest
Name | Type | Description |
---|---|---|
job_id | string |
CreateJobRunResponse
Name | Type | Description |
---|
DatabaseError
Name | Type | Description |
---|---|---|
errors | repeated string |
DeleteJobDestinationConnectionRequest
Name | Type | Description |
---|---|---|
destination_id | string |
DeleteJobDestinationConnectionResponse
Name | Type | Description |
---|
DeleteJobRequest
Name | Type | Description |
---|---|---|
id | string |
DeleteJobResponse
Name | Type | Description |
---|
DeleteJobRunRequest
Name | Type | Description |
---|---|---|
job_run_id | string | |
account_id | string |
DeleteJobRunResponse
Name | Type | Description |
---|
GcpCloudStorageDestinationConnectionOptions
Configuration for Google Cloud Storage Connection Job Options
Name | Type | Description |
---|
GenerateSourceOptions
Name | Type | Description |
---|---|---|
schemas | repeated GenerateSourceSchemaOption | |
fk_source_connection_id | optional string |
GenerateSourceSchemaOption
Name | Type | Description |
---|---|---|
schema | string | |
tables | repeated GenerateSourceTableOption |
GenerateSourceTableOption
Name | Type | Description |
---|---|---|
table | string | |
row_count | int64 |
GetJobNextRunsRequest
Name | Type | Description |
---|---|---|
job_id | string |
GetJobNextRunsResponse
Name | Type | Description |
---|---|---|
next_runs | JobNextRuns |
GetJobRecentRunsRequest
Name | Type | Description |
---|---|---|
job_id | string |
GetJobRecentRunsResponse
Name | Type | Description |
---|---|---|
recent_runs | repeated JobRecentRun |
GetJobRequest
Name | Type | Description |
---|---|---|
id | string |
GetJobResponse
Name | Type | Description |
---|---|---|
job | Job |
GetJobRunEventsRequest
Name | Type | Description |
---|---|---|
job_run_id | string | |
account_id | string |
GetJobRunEventsResponse
Name | Type | Description |
---|---|---|
events | repeated JobRunEvent | |
is_run_complete | bool |
GetJobRunLogsStreamRequest
Name | Type | Description |
---|---|---|
job_run_id | string | |
account_id | string | |
window | LogWindow | The time window in which to retrieve the logs |
should_tail | bool | Whether or not to tail the stream. Note: only works with k8s-pods and is not currently supported with Loki logs |
max_log_lines | optional int64 | Optionally provide a max log limit |
log_levels | repeated LogLevel | Provide a list of log levels to filter by. If any of these are UNSPECIFIED, all log levels are returned. |
GetJobRunLogsStreamResponse
Name | Type | Description |
---|---|---|
log_line | string | |
timestamp | optional google.protobuf.Timestamp |
GetJobRunRequest
Name | Type | Description |
---|---|---|
job_run_id | string | |
account_id | string |
GetJobRunResponse
Name | Type | Description |
---|---|---|
job_run | JobRun |
GetJobRunsRequest
Name | Type | Description |
---|---|---|
job_id | string | |
account_id | string |
GetJobRunsResponse
Name | Type | Description |
---|---|---|
job_runs | repeated JobRun |
GetJobStatusRequest
Name | Type | Description |
---|---|---|
job_id | string |
GetJobStatusResponse
Name | Type | Description |
---|---|---|
status | JobStatus |
GetJobStatusesRequest
Name | Type | Description |
---|---|---|
account_id | string |
GetJobStatusesResponse
Name | Type | Description |
---|---|---|
statuses | repeated JobStatusRecord |
GetJobsRequest
Name | Type | Description |
---|---|---|
account_id | string |
GetJobsResponse
Name | Type | Description |
---|---|---|
jobs | repeated Job |
IsJobNameAvailableRequest
Name | Type | Description |
---|---|---|
name | string | |
account_id | string |
IsJobNameAvailableResponse
Name | Type | Description |
---|---|---|
is_available | bool |
Job
Name | Type | Description |
---|---|---|
id | string | The unique identifier of the job |
created_by_user_id | string | |
created_at | google.protobuf.Timestamp | |
updated_by_user_id | string | |
updated_at | google.protobuf.Timestamp | |
name | string | The unique, friendly name of the job |
source | JobSource | |
destinations | repeated JobDestination | |
mappings | repeated JobMapping | |
cron_schedule | optional string | |
account_id | string | The account identifier that a job is associated with |
sync_options | ActivityOptions | Specify timeout and retry options for data synchronization activities Data sync activities are any piece of work that involves actually synchronizing data from a source to a destination For the data sync and generate jobs, this will be applied per table |
workflow_options | WorkflowOptions | Specify timeouts and other workflow options for the underlying temporal workflow |
virtual_foreign_keys | repeated VirtualForeignConstraint |
JobDestination
Name | Type | Description |
---|---|---|
connection_id | string | |
options | JobDestinationOptions | |
id | string |
JobDestinationOptions
Name | Type | Description |
---|---|---|
postgres_options | PostgresDestinationConnectionOptions | |
aws_s3_options | AwsS3DestinationConnectionOptions | |
mysql_options | MysqlDestinationConnectionOptions | |
mongodb_options | MongoDBDestinationConnectionOptions | |
gcp_cloudstorage_options | GcpCloudStorageDestinationConnectionOptions | Destination Connecton options for Google Cloud Storage |
JobMapping
Name | Type | Description |
---|---|---|
schema | string | |
table | string | |
column | string | |
transformer | JobMappingTransformer |
JobMappingTransformer
Name | Type | Description |
---|---|---|
source | TransformerSource | |
config | TransformerConfig |
JobNextRuns
Name | Type | Description |
---|---|---|
next_run_times | repeated google.protobuf.Timestamp |
JobRecentRun
Name | Type | Description |
---|---|---|
start_time | google.protobuf.Timestamp | |
job_run_id | string |
JobRun
Name | Type | Description |
---|---|---|
id | string | The id of the job run. This will currently be equivalent to the temporal workflow id |
job_id | string | The unique identifier of the job id this run is associated with |
name | string | The name of the job run. |
status | JobRunStatus | the status of the job run |
started_at | google.protobuf.Timestamp | A timestamp of when the run started |
completed_at | optional google.protobuf.Timestamp | Available if the run completed or has not yet been archived by the system |
pending_activities | repeated PendingActivity | Pending activities are only returned when retrieving a specific job run and will not be returned when requesting job runs in list format |
JobRunEvent
Name | Type | Description |
---|---|---|
id | int64 | |
type | string | |
start_time | google.protobuf.Timestamp | |
close_time | google.protobuf.Timestamp | |
metadata | JobRunEventMetadata | |
tasks | repeated JobRunEventTask |
JobRunEventMetadata
Name | Type | Description |
---|---|---|
sync_metadata | JobRunSyncMetadata |
JobRunEventTask
Name | Type | Description |
---|---|---|
id | int64 | |
type | string | |
event_time | google.protobuf.Timestamp | |
error | JobRunEventTaskError |
JobRunEventTaskError
Name | Type | Description |
---|---|---|
message | string | |
retry_state | string |
JobRunSyncMetadata
Name | Type | Description |
---|---|---|
schema | string | |
table | string |
JobSource
Name | Type | Description |
---|---|---|
options | JobSourceOptions |
JobSourceOptions
Name | Type | Description |
---|---|---|
postgres | PostgresSourceConnectionOptions | |
aws_s3 | AwsS3SourceConnectionOptions | |
mysql | MysqlSourceConnectionOptions | |
generate | GenerateSourceOptions | |
ai_generate | AiGenerateSourceOptions | |
mongodb | MongoDBSourceConnectionOptions |
JobSourceSqlSubetSchemas
Name | Type | Description |
---|---|---|
postgres_subset | PostgresSourceSchemaSubset | |
mysql_subset | MysqlSourceSchemaSubset |
JobStatusRecord
Name | Type | Description |
---|---|---|
job_id | string | |
status | JobStatus |
MongoDBDestinationConnectionOptions
Name | Type | Description |
---|
MongoDBSourceConnectionOptions
Name | Type | Description |
---|---|---|
connection_id | string |
MysqlDestinationConnectionOptions
Name | Type | Description |
---|---|---|
truncate_table | MysqlTruncateTableConfig | |
init_table_schema | bool | |
on_conflict | MysqlOnConflictConfig |
MysqlOnConflictConfig
Name | Type | Description |
---|---|---|
do_nothing | bool |
MysqlSourceConnectionOptions
Name | Type | Description |
---|---|---|
halt_on_new_column_addition | bool | |
schemas | repeated MysqlSourceSchemaOption | |
connection_id | string | |
subset_by_foreign_key_constraints | bool |
MysqlSourceSchemaOption
Name | Type | Description |
---|---|---|
schema | string | |
tables | repeated MysqlSourceTableOption |
MysqlSourceSchemaSubset
Name | Type | Description |
---|---|---|
mysql_schemas | repeated MysqlSourceSchemaOption |
MysqlSourceTableOption
Name | Type | Description |
---|---|---|
table | string | |
where_clause | optional string |
MysqlTruncateTableConfig
Name | Type | Description |
---|---|---|
truncate_before_insert | bool |
PauseJobRequest
Name | Type | Description |
---|---|---|
id | string | |
pause | bool | |
note | optional string |
PauseJobResponse
Name | Type | Description |
---|---|---|
job | Job |
PendingActivity
Name | Type | Description |
---|---|---|
status | ActivityStatus | |
activity_name | string | |
last_failure | optional ActivityFailure |
PostgresDestinationConnectionOptions
Name | Type | Description |
---|---|---|
truncate_table | PostgresTruncateTableConfig | |
init_table_schema | bool | |
on_conflict | PostgresOnConflictConfig |
PostgresOnConflictConfig
Name | Type | Description |
---|---|---|
do_nothing | bool |
PostgresSourceConnectionOptions
Name | Type | Description |
---|---|---|
halt_on_new_column_addition | bool | |
schemas | repeated PostgresSourceSchemaOption | |
connection_id | string | |
subset_by_foreign_key_constraints | bool |
PostgresSourceSchemaOption
Name | Type | Description |
---|---|---|
schema | string | |
tables | repeated PostgresSourceTableOption |
PostgresSourceSchemaSubset
Name | Type | Description |
---|---|---|
postgres_schemas | repeated PostgresSourceSchemaOption |
PostgresSourceTableOption
Name | Type | Description |
---|---|---|
table | string | |
where_clause | optional string |
PostgresTruncateTableConfig
Name | Type | Description |
---|---|---|
truncate_before_insert | bool | |
cascade | bool |
RetryPolicy
Defines the retry policy for an activity
Name | Type | Description |
---|---|---|
maximum_attempts | optional int32 | Maximum number of attempts. When exceeded the retries stop even if not expired yet. If not set or set to 0, it means unlimited, and rely on activity ScheduleToCloseTimeout to stop. |
SetJobSourceSqlConnectionSubsetsRequest
Name | Type | Description |
---|---|---|
id | string | |
schemas | JobSourceSqlSubetSchemas | |
subset_by_foreign_key_constraints | bool |
SetJobSourceSqlConnectionSubsetsResponse
Name | Type | Description |
---|---|---|
job | Job |
SetJobSyncOptionsRequest
Name | Type | Description |
---|---|---|
id | string | The unique identifier of the job |
sync_options | ActivityOptions | The sync options object. The entire object must be provided and will fully overwrite the previous result |
SetJobSyncOptionsResponse
Name | Type | Description |
---|---|---|
job | Job |
SetJobWorkflowOptionsRequest
Name | Type | Description |
---|---|---|
id | string | The unique identifier of the job |
worfklow_options | WorkflowOptions | The workflow options object. The entire object must be provided and will fully overwrite the previous result |
SetJobWorkflowOptionsResponse
Name | Type | Description |
---|---|---|
job | Job |
TerminateJobRunRequest
Name | Type | Description |
---|---|---|
job_run_id | string | |
account_id | string |
TerminateJobRunResponse
Name | Type | Description |
---|
UpdateJobDestinationConnectionRequest
Name | Type | Description |
---|---|---|
job_id | string | |
connection_id | string | |
options | JobDestinationOptions | |
destination_id | string |
UpdateJobDestinationConnectionResponse
Name | Type | Description |
---|---|---|
job | Job |
UpdateJobScheduleRequest
Name | Type | Description |
---|---|---|
id | string | |
cron_schedule | optional string |
UpdateJobScheduleResponse
Name | Type | Description |
---|---|---|
job | Job |
UpdateJobSourceConnectionRequest
Name | Type | Description |
---|---|---|
id | string | |
source | JobSource | |
mappings | repeated JobMapping | |
virtual_foreign_keys | repeated VirtualForeignConstraint |
UpdateJobSourceConnectionResponse
Name | Type | Description |
---|---|---|
job | Job |
ValidateJobMappingsRequest
Name | Type | Description |
---|---|---|
account_id | string | The unique account identifier that this job will be associated with |
mappings | repeated JobMapping | |
connection_id | string | |
virtual_foreign_keys | repeated VirtualForeignConstraint |
ValidateJobMappingsResponse
Name | Type | Description |
---|---|---|
column_errors | repeated ColumnError | |
database_errors | DatabaseError |
VirtualForeignConstraint
Name | Type | Description |
---|---|---|
schema | string | |
table | string | |
columns | repeated string | |
foreign_key | VirtualForeignKey |
VirtualForeignKey
Name | Type | Description |
---|---|---|
schema | string | |
table | string | |
columns | repeated string |
WorkflowOptions
Config that contains various timeouts that are configured in the underlying temporal workflow More options will come in the future as needed
Name | Type | Description |
---|---|---|
run_timeout | optional int64 | The timeout for a single workflow run. Measured in seconds |
Enums
ActivityStatus
Name | Number | Description |
---|---|---|
ACTIVITY_STATUS_UNSPECIFIED | 0 | |
ACTIVITY_STATUS_SCHEDULED | 1 | |
ACTIVITY_STATUS_STARTED | 2 | |
ACTIVITY_STATUS_CANCELED | 3 | |
ACTIVITY_STATUS_FAILED | 4 |
JobRunStatus
An enumeration of job run statuses.
Name | Number | Description |
---|---|---|
JOB_RUN_STATUS_UNSPECIFIED | 0 | if the job run status is unknown |
JOB_RUN_STATUS_PENDING | 1 | the run is pending and has not started yet |
JOB_RUN_STATUS_RUNNING | 2 | the run is currently in progress |
JOB_RUN_STATUS_COMPLETE | 3 | the run has successfully completed |
JOB_RUN_STATUS_ERROR | 4 | the run ended with an error |
JOB_RUN_STATUS_CANCELED | 5 | the run was cancelled |
JOB_RUN_STATUS_TERMINATED | 6 | the run was terminated |
JOB_RUN_STATUS_FAILED | 7 | the run ended in failure |
JOB_RUN_STATUS_TIMED_OUT | 8 | the run was ended pre-maturely due to timeout |
JobStatus
Name | Number | Description |
---|---|---|
JOB_STATUS_UNSPECIFIED | 0 | |
JOB_STATUS_ENABLED | 1 | |
JOB_STATUS_PAUSED | 3 | |
JOB_STATUS_DISABLED | 4 |
LogLevel
Name | Number | Description |
---|---|---|
LOG_LEVEL_UNSPECIFIED | 0 | |
LOG_LEVEL_DEBUG | 1 | |
LOG_LEVEL_INFO | 2 | |
LOG_LEVEL_WARN | 3 | |
LOG_LEVEL_ERROR | 4 |
LogWindow
Name | Number | Description |
---|---|---|
LOG_WINDOW_NO_TIME_UNSPECIFIED | 0 | |
LOG_WINDOW_FIFTEEN_MIN | 1 | |
LOG_WINDOW_ONE_HOUR | 2 | |
LOG_WINDOW_ONE_DAY | 3 |
Services
JobService
GetJobs
Method | GetJobs |
---|---|
Request | GetJobsRequest |
Response | GetJobsResponse |
Description |
GetJob
Method | GetJob |
---|---|
Request | GetJobRequest |
Response | GetJobResponse |
Description |
CreateJob
Method | CreateJob |
---|---|
Request | CreateJobRequest |
Response | CreateJobResponse |
Description |
DeleteJob
Method | DeleteJob |
---|---|
Request | DeleteJobRequest |
Response | DeleteJobResponse |
Description |
IsJobNameAvailable
Method | IsJobNameAvailable |
---|---|
Request | IsJobNameAvailableRequest |
Response | IsJobNameAvailableResponse |
Description |
UpdateJobSchedule
Method | UpdateJobSchedule |
---|---|
Request | UpdateJobScheduleRequest |
Response | UpdateJobScheduleResponse |
Description |
UpdateJobSourceConnection
Method | UpdateJobSourceConnection |
---|---|
Request | UpdateJobSourceConnectionRequest |
Response | UpdateJobSourceConnectionResponse |
Description |
SetJobSourceSqlConnectionSubsets
Method | SetJobSourceSqlConnectionSubsets |
---|---|
Request | SetJobSourceSqlConnectionSubsetsRequest |
Response | SetJobSourceSqlConnectionSubsetsResponse |
Description |
UpdateJobDestinationConnection
Method | UpdateJobDestinationConnection |
---|---|
Request | UpdateJobDestinationConnectionRequest |
Response | UpdateJobDestinationConnectionResponse |
Description |
DeleteJobDestinationConnection
Method | DeleteJobDestinationConnection |
---|---|
Request | DeleteJobDestinationConnectionRequest |
Response | DeleteJobDestinationConnectionResponse |
Description |
CreateJobDestinationConnections
Method | CreateJobDestinationConnections |
---|---|
Request | CreateJobDestinationConnectionsRequest |
Response | CreateJobDestinationConnectionsResponse |
Description |
PauseJob
Method | PauseJob |
---|---|
Request | PauseJobRequest |
Response | PauseJobResponse |
Description |
GetJobRecentRuns
Method | GetJobRecentRuns |
---|---|
Request | GetJobRecentRunsRequest |
Response | GetJobRecentRunsResponse |
Description | Returns a list of recently invoked job runs baseds on the Temporal cron scheduler. This will return a list of job runs that include archived runs |
GetJobNextRuns
Method | GetJobNextRuns |
---|---|
Request | GetJobNextRunsRequest |
Response | GetJobNextRunsResponse |
Description | Returns a list of runs that are scheduled for execution based on the Temporal cron scheduler. |
GetJobStatus
Method | GetJobStatus |
---|---|
Request | GetJobStatusRequest |
Response | GetJobStatusResponse |
Description |
GetJobStatuses
Method | GetJobStatuses |
---|---|
Request | GetJobStatusesRequest |
Response | GetJobStatusesResponse |
Description |
GetJobRuns
Method | GetJobRuns |
---|---|
Request | GetJobRunsRequest |
Response | GetJobRunsResponse |
Description | Returns a list of job runs by either account or job |
GetJobRunEvents
Method | GetJobRunEvents |
---|---|
Request | GetJobRunEventsRequest |
Response | GetJobRunEventsResponse |
Description |
GetJobRun
Method | GetJobRun |
---|---|
Request | GetJobRunRequest |
Response | GetJobRunResponse |
Description | Returns a specific job run, along with any of its pending activities |
DeleteJobRun
Method | DeleteJobRun |
---|---|
Request | DeleteJobRunRequest |
Response | DeleteJobRunResponse |
Description |
CreateJobRun
Method | CreateJobRun |
---|---|
Request | CreateJobRunRequest |
Response | CreateJobRunResponse |
Description |
CancelJobRun
Method | CancelJobRun |
---|---|
Request | CancelJobRunRequest |
Response | CancelJobRunResponse |
Description |
TerminateJobRun
Method | TerminateJobRun |
---|---|
Request | TerminateJobRunRequest |
Response | TerminateJobRunResponse |
Description |
GetJobRunLogsStream
Method | GetJobRunLogsStream |
---|---|
Request | GetJobRunLogsStreamRequest |
Response | GetJobRunLogsStreamResponse stream |
Description | Returns a stream of logs from the worker nodes that pertain to a specific job run |
SetJobWorkflowOptions
Method | SetJobWorkflowOptions |
---|---|
Request | SetJobWorkflowOptionsRequest |
Response | SetJobWorkflowOptionsResponse |
Description | Set any job workflow options. Must provide entire object as is it will fully override the previous configuration |
SetJobSyncOptions
Method | SetJobSyncOptions |
---|---|
Request | SetJobSyncOptionsRequest |
Response | SetJobSyncOptionsResponse |
Description | Set the job sync options. Must provide entire object as it will fully override the previous configuration |
ValidateJobMappings
Method | ValidateJobMappings |
---|---|
Request | ValidateJobMappingsRequest |
Response | ValidateJobMappingsResponse |
Description | validates that the jobmapping configured can run with table constraints |