Class PySparkStreamingJobProps
Properties for creating a Python Spark ETL job.
Inherited Members
Namespace: Amazon.CDK.AWS.Glue
Assembly: Amazon.CDK.Lib.dll
Syntax (csharp)
public class PySparkStreamingJobProps : IPySparkStreamingJobProps, ISparkJobProps, IJobPropsSyntax (vb)
Public Class PySparkStreamingJobProps Implements IPySparkStreamingJobProps, ISparkJobProps, IJobPropsRemarks
ExampleMetadata: infused
Examples
using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
Stack stack;
IRole role;
Code script;
new PySparkStreamingJob(stack, "ImportedJob", new PySparkStreamingJobProps { Role = role, Script = script });Synopsis
Constructors
| PySparkStreamingJobProps() | Properties for creating a Python Spark ETL job. |
Properties
| Connections | Connections (optional) List of connections to use for this Glue job Connections are used to connect to other AWS Service or resources within a VPC. |
| ContinuousLogging | Enables continuous logging with the specified props. |
| DefaultArguments | Default Arguments (optional) The default arguments for every run of this Glue job, specified as name-value pairs. |
| Description | Description (optional) Developer-specified description of the Glue job. |
| EnableMetrics | Enable profiling metrics for the Glue job. |
| EnableObservabilityMetrics | Enable observability metrics for the Glue job. |
| ExtraFiles | Additional files, such as configuration files that AWS Glue copies to the working directory of your script before executing it. |
| ExtraJars | Extra Jars S3 URL (optional) S3 URL where additional jar dependencies are located. |
| ExtraJarsFirst | Setting this value to true prioritizes the customer's extra JAR files in the classpath. |
| ExtraPythonFiles | Extra Python Files S3 URL (optional) S3 URL where additional python dependencies are located. |
| GlueVersion | Glue Version The version of Glue to use to execute this job. |
| JobName | Name of the Glue job (optional) Developer-specified name of the Glue job. |
| JobRunQueuingEnabled | Specifies whether job run queuing is enabled for the job runs for this job. |
| MaxConcurrentRuns | Max Concurrent Runs (optional) The maximum number of runs this Glue job can concurrently run. |
| MaxRetries | Max Retries (optional) Maximum number of retry attempts Glue performs if the job fails. |
| Role | IAM Role (required) IAM Role to use for Glue job execution Must be specified by the developer because the L2 doesn't have visibility into the actions the script(s) takes during the job execution The role must trust the Glue service principal (glue.amazonaws.com) and be granted sufficient permissions. |
| Script | Script Code Location (required) Script to run when the Glue job executes. |
| SecurityConfiguration | Security Configuration (optional) Defines the encryption options for the Glue job. |
| SparkUI | Enables the Spark UI debugging and monitoring with the specified props. |
| Tags | Tags (optional) A list of key:value pairs of tags to apply to this Glue job resources. |
| Timeout | Timeout (optional) The maximum time that a job run can consume resources before it is terminated and enters TIMEOUT status. |
| WorkerConfiguration | The worker type and the number of workers allocated when a job runs. |
Constructors
PySparkStreamingJobProps()
Properties for creating a Python Spark ETL job.
public PySparkStreamingJobProps()Remarks
ExampleMetadata: infused
Examples
using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
Stack stack;
IRole role;
Code script;
new PySparkStreamingJob(stack, "ImportedJob", new PySparkStreamingJobProps { Role = role, Script = script });Properties
Connections
Connections (optional) List of connections to use for this Glue job Connections are used to connect to other AWS Service or resources within a VPC.
public IConnection[]? Connections { get; set; }Property Value
Remarks
Default: [] - no connections are added to the job
ContinuousLogging
Enables continuous logging with the specified props.
public IContinuousLoggingProps? ContinuousLogging { get; set; }Property Value
Remarks
Default: - continuous logging is enabled.
DefaultArguments
Default Arguments (optional) The default arguments for every run of this Glue job, specified as name-value pairs.
public IDictionary<string, string>? DefaultArguments { get; set; }Property Value
Remarks
This map is the escape hatch for Glue job arguments that this construct does not model. It
MUST NOT be used to set arguments that already have a dedicated prop — configure those through
the corresponding prop instead (continuousLogging, enableMetrics,
enableObservabilityMetrics, sparkUI, className, extraJars, extraJarsFirst,
extraPythonFiles, extraFiles). Passing a construct-managed argument (e.g.
--enable-continuous-cloudwatch-log, --enable-metrics, --enable-spark-ui,
--job-language) or a Glue-reserved argument (--debug, --mode, --JOB_NAME, --endpoint)
here throws at synthesis time, so there is exactly one way to express each intent.
Also note that these are emitted verbatim into the CloudFormation template, so avoid placing secrets here in plaintext. Pass secrets to the job at runtime through AWS Secrets Manager instead. A synthesis-time warning is emitted when an argument key looks like a credential and holds a plaintext literal.
Default: - no arguments
See: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html for a list of reserved parameters
Description
Description (optional) Developer-specified description of the Glue job.
public string? Description { get; set; }Property Value
Remarks
Default: - no value
EnableMetrics
Enable profiling metrics for the Glue job.
public bool? EnableMetrics { get; set; }Property Value
bool?
Remarks
When enabled, adds '--enable-metrics' to job arguments.
Default: true
EnableObservabilityMetrics
Enable observability metrics for the Glue job.
public bool? EnableObservabilityMetrics { get; set; }Property Value
bool?
Remarks
When enabled, adds '--enable-observability-metrics': 'true' to job arguments.
Default: true
ExtraFiles
Additional files, such as configuration files that AWS Glue copies to the working directory of your script before executing it.
public Code[]? ExtraFiles { get; set; }Property Value
Code[]
Remarks
Default: - no extra files specified.
ExtraJars
Extra Jars S3 URL (optional) S3 URL where additional jar dependencies are located.
public Code[]? ExtraJars { get; set; }Property Value
Code[]
Remarks
Default: - no extra jar files
ExtraJarsFirst
Setting this value to true prioritizes the customer's extra JAR files in the classpath.
public bool? ExtraJarsFirst { get; set; }Property Value
bool?
Remarks
Default: false - priority is not given to user-provided jars
See: --user-jars-first in https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html
ExtraPythonFiles
Extra Python Files S3 URL (optional) S3 URL where additional python dependencies are located.
public Code[]? ExtraPythonFiles { get; set; }Property Value
Code[]
Remarks
Default: - no extra files
GlueVersion
Glue Version The version of Glue to use to execute this job.
public GlueVersion? GlueVersion { get; set; }Property Value
Remarks
Default: - determined by the job type: 4.0 for ETL and Streaming, 5.0 for Flex, 3.0 for Python Shell
JobName
Name of the Glue job (optional) Developer-specified name of the Glue job.
public string? JobName { get; set; }Property Value
Remarks
Default: - a name is automatically generated
JobRunQueuingEnabled
Specifies whether job run queuing is enabled for the job runs for this job.
public bool? JobRunQueuingEnabled { get; set; }Property Value
bool?
Remarks
A value of true means job run queuing is enabled for the job runs. If false or not populated, the job runs will not be considered for queueing. If this field does not match the value set in the job run, then the value from the job run field will be used. This property must be set to false for flex jobs. If this property is enabled, maxRetries must be set to zero.
Default: - no job run queuing
MaxConcurrentRuns
Max Concurrent Runs (optional) The maximum number of runs this Glue job can concurrently run.
public double? MaxConcurrentRuns { get; set; }Property Value
Remarks
An error is returned when this threshold is reached. The maximum value you can specify is controlled by a service limit.
Default: 1
MaxRetries
Max Retries (optional) Maximum number of retry attempts Glue performs if the job fails.
public double? MaxRetries { get; set; }Property Value
Remarks
Default: 0
Role
IAM Role (required) IAM Role to use for Glue job execution Must be specified by the developer because the L2 doesn't have visibility into the actions the script(s) takes during the job execution The role must trust the Glue service principal (glue.amazonaws.com) and be granted sufficient permissions.
public IRole Role { get; set; }Property Value
Remarks
Script
Script Code Location (required) Script to run when the Glue job executes.
public Code Script { get; set; }Property Value
Remarks
Can be uploaded from the local directory structure using fromAsset or referenced via S3 location using fromBucket
SecurityConfiguration
Security Configuration (optional) Defines the encryption options for the Glue job.
public ISecurityConfiguration? SecurityConfiguration { get; set; }Property Value
Remarks
Default: - no security configuration.
SparkUI
Enables the Spark UI debugging and monitoring with the specified props.
public ISparkUIProps? SparkUI { get; set; }Property Value
Remarks
Default: - Spark UI debugging and monitoring is disabled.
Tags
Tags (optional) A list of key:value pairs of tags to apply to this Glue job resources.
public IDictionary<string, string>? Tags { get; set; }Property Value
Remarks
Default: {} - no tags
Timeout
Timeout (optional) The maximum time that a job run can consume resources before it is terminated and enters TIMEOUT status.
public Duration? Timeout { get; set; }Property Value
Remarks
Specified in minutes.
Default: - no value set; Glue applies its service default (2880 minutes for non-streaming jobs)
WorkerConfiguration
The worker type and the number of workers allocated when a job runs.
public IWorkerConfiguration? WorkerConfiguration { get; set; }Property Value
Remarks
Default: - the job runs with the G_1X worker type and 10 workers.