diff --git a/CONFIGURE.md b/CONFIGURE.md index e723dad68e..061f3232da 100644 --- a/CONFIGURE.md +++ b/CONFIGURE.md @@ -142,6 +142,7 @@ Remote Processing Groups: Besides YAML configuration format, MiNiFi C++ also supports JSON configuration. To see different uses cases in both formats, please refer to the [examples page](examples/README.md) for flow config examples. **NOTE:** Make sure to specify id for each component (Processor, Connection, Controller, RPG etc.) to make sure that Apache MiNiFi C++ can reload the state after a process restart. The id should be unique in the flow configuration. +**NOTE:** Make sure to terminate all outgoing connections of processors, ports, funnels, either by connecting them to another component or auto-terminating them, otherwise the flow validation fails on startup. ### Parameter Contexts diff --git a/core-framework/include/core/ProcessorImpl.h b/core-framework/include/core/ProcessorImpl.h index 8c385936e2..4f73843f87 100644 --- a/core-framework/include/core/ProcessorImpl.h +++ b/core-framework/include/core/ProcessorImpl.h @@ -84,6 +84,8 @@ class ProcessorImpl : public virtual ProcessorApi { void setSupportedProperties(std::span properties); void setSupportedProperties(std::span properties); + void addAutoTerminatedRelationship(const RelationshipDefinition& relationship); + virtual void initialize() {} void onTrigger(ProcessContext&, ProcessSession&) override {} diff --git a/core-framework/src/core/ProcessorImpl.cpp b/core-framework/src/core/ProcessorImpl.cpp index 3946242fe3..06fcc5f920 100644 --- a/core-framework/src/core/ProcessorImpl.cpp +++ b/core-framework/src/core/ProcessorImpl.cpp @@ -84,6 +84,11 @@ void ProcessorImpl::setSupportedRelationships(std::spansetSupportedRelationships(relationships); } +void ProcessorImpl::addAutoTerminatedRelationship(const RelationshipDefinition& relationship) { + gsl_Expects(descriptor_); + descriptor_->addAutoTerminatedRelationship(relationship); +} + void ProcessorImpl::setSupportedProperties(std::span properties) { gsl_Expects(descriptor_); descriptor_->setSupportedProperties(properties); diff --git a/examples/couchbase_mtls_authentication.json b/examples/couchbase_mtls_authentication.json index 7ac14df254..a7637924c6 100644 --- a/examples/couchbase_mtls_authentication.json +++ b/examples/couchbase_mtls_authentication.json @@ -69,6 +69,6 @@ "success" ] } - ], + ] } } diff --git a/examples/process_data_with_scripts.json b/examples/process_data_with_scripts.json index 12d2b981e6..2be2ddc6ef 100644 --- a/examples/process_data_with_scripts.json +++ b/examples/process_data_with_scripts.json @@ -17,7 +17,8 @@ "Script File": "../../examples/scripts/python/reverse_flow_file_content.py" }, "autoTerminatedRelationships": [ - "failure" + "failure", + "original" ] }, { diff --git a/examples/process_data_with_scripts.yml b/examples/process_data_with_scripts.yml index 552f579880..b979a438fe 100644 --- a/examples/process_data_with_scripts.yml +++ b/examples/process_data_with_scripts.yml @@ -28,6 +28,7 @@ Processors: run duration nanos: 0 auto-terminated relationships list: - failure + - original Properties: Module Directory: Script Body: diff --git a/extensions/aws/tests/features/kinesis.feature b/extensions/aws/tests/features/kinesis.feature index 03823ab78c..5b4e6474f1 100644 --- a/extensions/aws/tests/features/kinesis.feature +++ b/extensions/aws/tests/features/kinesis.feature @@ -38,6 +38,7 @@ Feature: Sending data from MiNiFi-C++ to an AWS Kinesis server And the "success" relationship of the PutKinesisStream processor is connected to the PutFile And the "failure" relationship of the PutKinesisStream processor is connected to the PutKinesisStream And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When both instances start up diff --git a/extensions/aws/tests/features/s3.feature b/extensions/aws/tests/features/s3.feature index ffdbfa52c2..df8a246fde 100644 --- a/extensions/aws/tests/features/s3.feature +++ b/extensions/aws/tests/features/s3.feature @@ -28,6 +28,7 @@ Feature: Sending data from MiNiFi-C++ to an AWS server And the "success" relationship of the PutS3Object processor is connected to the PutFile And the "failure" relationship of the PutS3Object processor is connected to the PutS3Object And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And the s3 server starts up @@ -53,6 +54,7 @@ Feature: Sending data from MiNiFi-C++ to an AWS server And the "success" relationship of the PutS3Object processor is connected to the PutFile And the "failure" relationship of the PutS3Object processor is connected to the PutS3Object And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And the s3 server starts up And the http proxy server is set up @@ -72,7 +74,9 @@ Feature: Sending data from MiNiFi-C++ to an AWS server And the "success" relationship of the GetFile processor is connected to the PutS3Object And the "success" relationship of the PutS3Object processor is connected to the PutFile And a ProxyConfigurationService controller service is set up with HTTP proxy configuration + And PutS3Object's failure relationship is auto-terminated And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And the s3 server starts up And the http proxy server is set up @@ -95,7 +99,9 @@ Feature: Sending data from MiNiFi-C++ to an AWS server | PutS3Object | success | DeleteS3Object | | PutS3Object | failure | PutS3Object | | DeleteS3Object | success | PutFile | + | DeleteS3Object | failure | auto-terminated | | PutFile | success | auto-terminated | + | PutFile | failure | auto-terminated | And the s3 server starts up @@ -111,6 +117,9 @@ Feature: Sending data from MiNiFi-C++ to an AWS server And a PutFile processor with the "Directory" property set to "/tmp/output" And the "success" relationship of the GetFile processor is connected to the DeleteS3Object And the "success" relationship of the DeleteS3Object processor is connected to the PutFile + And DeleteS3Object's failure relationship is auto-terminated + And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And the s3 server starts up @@ -139,7 +148,9 @@ Feature: Sending data from MiNiFi-C++ to an AWS server | PutS3Object | failure | PutS3Object | | PutS3Object | success | DeleteS3Object | | DeleteS3Object | success | PutFile | + | DeleteS3Object | failure | auto-terminated | | PutFile | success | auto-terminated | + | PutFile | failure | auto-terminated | And the s3 server starts up And the http proxy server is set up @@ -162,9 +173,13 @@ Feature: Sending data from MiNiFi-C++ to an AWS server | source name | relationship name | destination name | | GetFile | success | PutS3Object | | PutS3Object | success | DeleteS3Object | + | PutS3Object | failure | auto-terminated | | DeleteS3Object | success | PutFile | + | DeleteS3Object | failure | auto-terminated | + | PutFile | success | auto-terminated | + | PutFile | failure | auto-terminated | + And a ProxyConfigurationService controller service is set up with HTTP proxy configuration - And PutFile's success relationship is auto-terminated And the s3 server starts up And the http proxy server is set up @@ -186,6 +201,7 @@ Feature: Sending data from MiNiFi-C++ to an AWS server | source name | relationship name | destination name | | GenerateFlowFile | success | FetchS3Object | | FetchS3Object | success | PutFile | + | FetchS3Object | failure | auto-terminated | | PutFile | success | auto-terminated | | PutFile | failure | auto-terminated | | GetFile | success | PutS3Object | @@ -218,6 +234,11 @@ Feature: Sending data from MiNiFi-C++ to an AWS server | source name | relationship name | destination name | | GenerateFlowFile | success | FetchS3Object | | FetchS3Object | success | PutFile | + | FetchS3Object | failure | auto-terminated | + | PutFile | failure | auto-terminated | + | PutFile | success | auto-terminated | + | PutS3Object | success | auto-terminated | + | PutS3Object | failure | auto-terminated | And the s3 server starts up And the http proxy server is set up @@ -233,6 +254,7 @@ Feature: Sending data from MiNiFi-C++ to an AWS server And a PutS3Object processor set up to communicate with an s3 server And the "success" relationship of the GetFile processor is connected to the PutS3Object And PutS3Object's success relationship is auto-terminated + And PutS3Object's failure relationship is auto-terminated Given a GenerateFlowFile processor with the "File Size" property set to "1 kB" And a FetchS3Object processor set up to communicate with the same s3 server @@ -242,8 +264,10 @@ Feature: Sending data from MiNiFi-C++ to an AWS server | source name | relationship name | destination name | | GenerateFlowFile | success | FetchS3Object | | FetchS3Object | success | PutFile | + | FetchS3Object | failure | auto-terminated | + | PutFile | success | auto-terminated | + | PutFile | failure | auto-terminated | And a ProxyConfigurationService controller service is set up with HTTP proxy configuration - And PutFile's success relationship is auto-terminated And the s3 server starts up And the http proxy server is set up @@ -266,6 +290,10 @@ Feature: Sending data from MiNiFi-C++ to an AWS server Given a ListS3 processor set up to communicate with the same s3 server And a PutFile processor with the "Directory" property set to "/tmp/output" And the "success" relationship of the ListS3 processor is connected to the PutFile + And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated + And PutS3Object's success relationship is auto-terminated + And PutS3Object's failure relationship is auto-terminated And the s3 server starts up @@ -278,6 +306,8 @@ Feature: Sending data from MiNiFi-C++ to an AWS server And a directory at "/tmp/input" has a file with the content "test" And a PutS3Object processor set up to communicate with an s3 server And the "success" relationship of the GetFile processor is connected to the PutS3Object + And PutS3Object's success relationship is auto-terminated + And PutS3Object's failure relationship is auto-terminated Given a ListS3 processor set up to communicate with the same s3 server And these processor properties are set @@ -288,6 +318,8 @@ Feature: Sending data from MiNiFi-C++ to an AWS server | ListS3 | Proxy Password | test101 | And a PutFile processor with the "Directory" property set to "/tmp/output" And the "success" relationship of the ListS3 processor is connected to the PutFile + And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And the s3 server starts up And the http proxy server is set up @@ -303,6 +335,7 @@ Feature: Sending data from MiNiFi-C++ to an AWS server And a PutS3Object processor set up to communicate with an s3 server And the "success" relationship of the GetFile processor is connected to the PutS3Object And PutS3Object's success relationship is auto-terminated + And PutS3Object's failure relationship is auto-terminated Given a ListS3 processor set up to communicate with the same s3 server And the "Proxy Configuration Service" property of the ListS3 processor is set to "ProxyConfigurationService" @@ -310,6 +343,7 @@ Feature: Sending data from MiNiFi-C++ to an AWS server And the "success" relationship of the ListS3 processor is connected to the PutFile And a ProxyConfigurationService controller service is set up with HTTP proxy configuration And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And the s3 server starts up And the http proxy server is set up @@ -328,6 +362,11 @@ Feature: Sending data from MiNiFi-C++ to an AWS server And the "success" relationship of the GetFile processor is connected to the PutS3Object And a PutFile processor with the "Directory" property set to "/tmp/output" And the "success" relationship of the PutS3Object processor is connected to the PutFile + And PutS3Object's success relationship is auto-terminated + And PutS3Object's failure relationship is auto-terminated + And PutFile's failure relationship is auto-terminated + And PutFile's success relationship is auto-terminated + And the s3 server starts up When the MiNiFi instance starts up @@ -351,6 +390,9 @@ Feature: Sending data from MiNiFi-C++ to an AWS server And the "success" relationship of the GetFile processor is connected to the PutS3Object And a PutFile processor with the "Directory" property set to "/tmp/output" And the "success" relationship of the PutS3Object processor is connected to the PutFile + And PutS3Object's failure relationship is auto-terminated + And PutFile's failure relationship is auto-terminated + And PutFile's success relationship is auto-terminated And the s3 server starts up And the http proxy server is set up diff --git a/extensions/azure/tests/features/azure_storage.feature b/extensions/azure/tests/features/azure_storage.feature index 5ac7fc4a58..c6f518cda1 100644 --- a/extensions/azure/tests/features/azure_storage.feature +++ b/extensions/azure/tests/features/azure_storage.feature @@ -28,6 +28,7 @@ Feature: Sending data from MiNiFi-C++ to an Azure storage server And the "success" relationship of the PutAzureBlobStorage processor is connected to the PutFile And the "failure" relationship of the PutAzureBlobStorage processor is connected to the PutAzureBlobStorage And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And an Azure storage server is set up When the MiNiFi instance starts up @@ -40,6 +41,8 @@ Feature: Sending data from MiNiFi-C++ to an Azure storage server And a DeleteAzureBlobStorage processor set up to communicate with an Azure blob storage And the "Blob" property of the DeleteAzureBlobStorage processor is set to "test" And the "success" relationship of the GenerateFlowFile processor is connected to the DeleteAzureBlobStorage + And DeleteAzureBlobStorage's success relationship is auto-terminated + And DeleteAzureBlobStorage's failure relationship is auto-terminated And an Azure storage server is set up And test blob "test" is created on Azure blob storage @@ -54,6 +57,7 @@ Feature: Sending data from MiNiFi-C++ to an Azure storage server And the "Delete Snapshots Option" property of the DeleteAzureBlobStorage processor is set to "Include Snapshots" And the "success" relationship of the GenerateFlowFile processor is connected to the DeleteAzureBlobStorage And DeleteAzureBlobStorage's success relationship is auto-terminated + And DeleteAzureBlobStorage's failure relationship is auto-terminated And an Azure storage server is set up And test blob "test" is created on Azure blob storage with a snapshot @@ -67,6 +71,7 @@ Feature: Sending data from MiNiFi-C++ to an Azure storage server And the "Delete Snapshots Option" property of the DeleteAzureBlobStorage processor is set to "Delete Snapshots Only" And the "success" relationship of the GenerateFlowFile processor is connected to the DeleteAzureBlobStorage And DeleteAzureBlobStorage's success relationship is auto-terminated + And DeleteAzureBlobStorage's failure relationship is auto-terminated And an Azure storage server is set up And test blob "test" is created on Azure blob storage with a snapshot @@ -84,7 +89,9 @@ Feature: Sending data from MiNiFi-C++ to an Azure storage server And a PutFile processor with the "Directory" property set to "/tmp/output" And the "success" relationship of the GetFile processor is connected to the FetchAzureBlobStorage And the "success" relationship of the FetchAzureBlobStorage processor is connected to the PutFile + And FetchAzureBlobStorage's failure relationship is auto-terminated And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And an Azure storage server is set up When the MiNiFi instance starts up @@ -119,6 +126,7 @@ Feature: Sending data from MiNiFi-C++ to an Azure storage server And the "success" relationship of the PutAzureBlobStorage processor is connected to the PutFile And the "failure" relationship of the PutAzureBlobStorage processor is connected to the PutAzureBlobStorage And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And a ProxyConfigurationService controller service is set up with HTTP proxy configuration And an Azure storage server is set up @@ -137,6 +145,7 @@ Feature: Sending data from MiNiFi-C++ to an Azure storage server And the "Proxy Configuration Service" property of the DeleteAzureBlobStorage processor is set to "ProxyConfigurationService" And the "success" relationship of the GenerateFlowFile processor is connected to the DeleteAzureBlobStorage And DeleteAzureBlobStorage's success relationship is auto-terminated + And DeleteAzureBlobStorage's failure relationship is auto-terminated And a ProxyConfigurationService controller service is set up with HTTP proxy configuration And an Azure storage server is set up @@ -161,6 +170,8 @@ Feature: Sending data from MiNiFi-C++ to an Azure storage server And the "success" relationship of the GetFile processor is connected to the FetchAzureBlobStorage And the "success" relationship of the FetchAzureBlobStorage processor is connected to the PutFile And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated + And FetchAzureBlobStorage's failure relationship is auto-terminated And a ProxyConfigurationService controller service is set up with HTTP proxy configuration And an Azure storage server is set up diff --git a/extensions/civetweb/tests/features/http.feature b/extensions/civetweb/tests/features/http.feature index f667ca6d9c..61588c74e7 100644 --- a/extensions/civetweb/tests/features/http.feature +++ b/extensions/civetweb/tests/features/http.feature @@ -25,6 +25,7 @@ Feature: Sending data using InvokeHTTP to a receiver using ListenHTTP And PutFile is EVENT_DRIVEN in the "secondary" flow And in the "secondary" flow the "success" relationship of the ListenHTTP processor is connected to the PutFile And PutFile's success relationship is auto-terminated in the "secondary" flow + And PutFile's failure relationship is auto-terminated in the "secondary" flow And a GetFile processor with the "Input Directory" property set to "/tmp/input" And the "Keep Source File" property of the GetFile processor is set to "true" @@ -36,6 +37,8 @@ Feature: Sending data using InvokeHTTP to a receiver using ListenHTTP And InvokeHTTP's success relationship is auto-terminated And InvokeHTTP's response relationship is auto-terminated And InvokeHTTP's failure relationship is auto-terminated + And InvokeHTTP's no retry relationship is auto-terminated + And InvokeHTTP's retry relationship is auto-terminated When both instances start up Then in the "secondary" container at least one file with the content "test" is placed in the "/tmp/output" directory in less than 60 seconds @@ -46,6 +49,7 @@ Feature: Sending data using InvokeHTTP to a receiver using ListenHTTP And PutFile is EVENT_DRIVEN in the "minifi-listen" flow And in the "minifi-listen" flow the "success" relationship of the ListenHTTP processor is connected to the PutFile And PutFile's success relationship is auto-terminated in the "minifi-listen" flow + And PutFile's failure relationship is auto-terminated in the "minifi-listen" flow And the http proxy server is set up And a GetFile processor with the "Input Directory" property set to "/tmp/input" @@ -64,6 +68,8 @@ Feature: Sending data using InvokeHTTP to a receiver using ListenHTTP And InvokeHTTP's success relationship is auto-terminated And InvokeHTTP's response relationship is auto-terminated And InvokeHTTP's failure relationship is auto-terminated + And InvokeHTTP's no retry relationship is auto-terminated + And InvokeHTTP's retry relationship is auto-terminated When all instances start up Then in the "minifi-listen" container at least one file with the content "test" is placed in the "/tmp/output" directory in less than 60 seconds @@ -75,6 +81,7 @@ Feature: Sending data using InvokeHTTP to a receiver using ListenHTTP And PutFile is EVENT_DRIVEN in the "secondary" flow And in the "secondary" flow the "success" relationship of the ListenHTTP processor is connected to the PutFile And PutFile's success relationship is auto-terminated in the "secondary" flow + And PutFile's failure relationship is auto-terminated in the "secondary" flow And a GetFile processor with the "Input Directory" property set to "/tmp/input" And the "Keep Source File" property of the GetFile processor is set to "true" @@ -86,9 +93,12 @@ Feature: Sending data using InvokeHTTP to a receiver using ListenHTTP And the "HTTP Method" property of the InvokeHTTP processor is set to "POST" And the "success" relationship of the GetFile processor is connected to the HashContent And the "success" relationship of the HashContent processor is connected to the InvokeHTTP + And HashContent's failure relationship is auto-terminated And InvokeHTTP's success relationship is auto-terminated And InvokeHTTP's response relationship is auto-terminated And InvokeHTTP's failure relationship is auto-terminated + And InvokeHTTP's no retry relationship is auto-terminated + And InvokeHTTP's retry relationship is auto-terminated When both instances start up Then in the "secondary" container at least one file with the content "test" is placed in the "/tmp/output" directory in less than 60 seconds @@ -99,6 +109,7 @@ Feature: Sending data using InvokeHTTP to a receiver using ListenHTTP And PutFile is EVENT_DRIVEN in the "secondary" flow And in the "secondary" flow the "success" relationship of the ListenHTTP processor is connected to the PutFile And PutFile's success relationship is auto-terminated in the "secondary" flow + And PutFile's failure relationship is auto-terminated in the "secondary" flow And a GetFile processor with the "Input Directory" property set to "/tmp/input" And the "Keep Source File" property of the GetFile processor is set to "true" @@ -111,6 +122,8 @@ Feature: Sending data using InvokeHTTP to a receiver using ListenHTTP And InvokeHTTP's success relationship is auto-terminated And InvokeHTTP's response relationship is auto-terminated And InvokeHTTP's failure relationship is auto-terminated + And InvokeHTTP's no retry relationship is auto-terminated + And InvokeHTTP's retry relationship is auto-terminated When both instances start up Then in the "secondary" container at least one empty file is placed in the "/tmp/output" directory in less than 60 seconds @@ -132,6 +145,8 @@ Feature: Sending data using InvokeHTTP to a receiver using ListenHTTP And InvokeHTTP's success relationship is auto-terminated And InvokeHTTP's response relationship is auto-terminated And InvokeHTTP's failure relationship is auto-terminated + And InvokeHTTP's no retry relationship is auto-terminated + And InvokeHTTP's retry relationship is auto-terminated When both instances start up Then in the "nifi" container at least one empty file is placed in the "/tmp/output" directory in less than 60 seconds @@ -142,6 +157,7 @@ Feature: Sending data using InvokeHTTP to a receiver using ListenHTTP And PutFile is EVENT_DRIVEN in the "secondary" flow And in the "secondary" flow the "success" relationship of the ListenHTTP processor is connected to the PutFile And PutFile's success relationship is auto-terminated in the "secondary" flow + And PutFile's failure relationship is auto-terminated in the "secondary" flow And a GenerateFlowFile processor with the "File Size" property set to "10 MB" And the scheduling period of the GenerateFlowFile processor is set to "30 sec" @@ -155,6 +171,8 @@ Feature: Sending data using InvokeHTTP to a receiver using ListenHTTP And InvokeHTTP's success relationship is auto-terminated And InvokeHTTP's response relationship is auto-terminated And InvokeHTTP's failure relationship is auto-terminated + And InvokeHTTP's no retry relationship is auto-terminated + And InvokeHTTP's retry relationship is auto-terminated When both instances start up Then in the "secondary" container at least one file with minimum size of "1 MB" is placed in the "/tmp/output" directory in less than 60 seconds @@ -170,6 +188,7 @@ Feature: Sending data using InvokeHTTP to a receiver using ListenHTTP And ListenHTTP is EVENT_DRIVEN in the "secondary" flow And in the "secondary" flow the "success" relationship of the GenerateFlowFile processor is connected to the UpdateAttribute And in the "secondary" flow the "success" relationship of the UpdateAttribute processor is connected to the ListenHTTP + And UpdateAttribute's failure relationship is auto-terminated in the "secondary" flow And ListenHTTP's success relationship is auto-terminated in the "secondary" flow And a InvokeHTTP processor with the "Remote URL" property set to "http://secondary-${scenario_id}:8080/contentListener&testfile" @@ -185,6 +204,10 @@ Feature: Sending data using InvokeHTTP to a receiver using ListenHTTP And InvokeHTTP's success relationship is auto-terminated And InvokeHTTP's response relationship is auto-terminated And InvokeHTTP's failure relationship is auto-terminated + And InvokeHTTP's no retry relationship is auto-terminated + And InvokeHTTP's retry relationship is auto-terminated + And PutFile's failure relationship is auto-terminated + And PutFile's success relationship is auto-terminated When both instances start up Then at least one file with minimum size of "10 MB" is placed in the "/tmp/output" directory in less than 60 seconds diff --git a/extensions/civetweb/tests/features/https.feature b/extensions/civetweb/tests/features/https.feature index ff31824abf..94856f54c2 100644 --- a/extensions/civetweb/tests/features/https.feature +++ b/extensions/civetweb/tests/features/https.feature @@ -23,6 +23,7 @@ Feature: Transfer data from and to MiNiFi using HTTPS And PutFile is EVENT_DRIVEN in the "server" flow And in the "server" flow the "success" relationship of the ListenHTTP processor is connected to the PutFile And PutFile's success relationship is auto-terminated in the "server" flow + And PutFile's failure relationship is auto-terminated in the "server" flow And a GenerateFlowFile processor with the "Data Format" property set to "Text" And the "Unique FlowFiles" property of the GenerateFlowFile processor is set to "false" @@ -34,6 +35,8 @@ Feature: Transfer data from and to MiNiFi using HTTPS And InvokeHTTP's success relationship is auto-terminated And InvokeHTTP's response relationship is auto-terminated And InvokeHTTP's failure relationship is auto-terminated + And InvokeHTTP's no retry relationship is auto-terminated + And InvokeHTTP's retry relationship is auto-terminated When both instances start up Then in the "server" container at least one file with the content "Lorem ipsum dolor sit amet" is placed in the "/tmp/output" directory in less than 10 seconds @@ -44,6 +47,7 @@ Feature: Transfer data from and to MiNiFi using HTTPS And a PutFile processor with the "Directory" property set to "/tmp/output" in the "server" flow And in the "server" flow the "success" relationship of the ListenHTTP processor is connected to the PutFile And PutFile's success relationship is auto-terminated in the "server" flow + And PutFile's failure relationship is auto-terminated in the "server" flow And a GenerateFlowFile processor with the "Data Format" property set to "Text" And the "Unique FlowFiles" property of the GenerateFlowFile processor is set to "false" @@ -55,6 +59,8 @@ Feature: Transfer data from and to MiNiFi using HTTPS And InvokeHTTP's success relationship is auto-terminated And InvokeHTTP's response relationship is auto-terminated And InvokeHTTP's failure relationship is auto-terminated + And InvokeHTTP's no retry relationship is auto-terminated + And InvokeHTTP's retry relationship is auto-terminated When both instances start up Then in the "server" container no files are placed in the "/tmp/output" directory in 10s of running time @@ -68,6 +74,7 @@ Feature: Transfer data from and to MiNiFi using HTTPS And PutFile is EVENT_DRIVEN in the "server" flow And in the "server" flow the "success" relationship of the ListenHTTP processor is connected to the PutFile And PutFile's success relationship is auto-terminated in the "server" flow + And PutFile's failure relationship is auto-terminated in the "server" flow And a GenerateFlowFile processor with the "Data Format" property set to "Text" And the "Unique FlowFiles" property of the GenerateFlowFile processor is set to "false" @@ -80,6 +87,8 @@ Feature: Transfer data from and to MiNiFi using HTTPS And InvokeHTTP's success relationship is auto-terminated And InvokeHTTP's response relationship is auto-terminated And InvokeHTTP's failure relationship is auto-terminated + And InvokeHTTP's no retry relationship is auto-terminated + And InvokeHTTP's retry relationship is auto-terminated When both instances start up Then in the "server" container at least one file with the content "ut labore et dolore magna aliqua" is placed in the "/tmp/output" directory in less than 10s @@ -90,6 +99,7 @@ Feature: Transfer data from and to MiNiFi using HTTPS And a PutFile processor with the "Directory" property set to "/tmp/output" in the "server" flow And in the "server" flow the "success" relationship of the ListenHTTP processor is connected to the PutFile And PutFile's success relationship is auto-terminated in the "server" flow + And PutFile's failure relationship is auto-terminated in the "server" flow And a GenerateFlowFile processor with the "Data Format" property set to "Text" And the "Unique FlowFiles" property of the GenerateFlowFile processor is set to "false" @@ -102,6 +112,8 @@ Feature: Transfer data from and to MiNiFi using HTTPS And InvokeHTTP's success relationship is auto-terminated And InvokeHTTP's response relationship is auto-terminated And InvokeHTTP's failure relationship is auto-terminated + And InvokeHTTP's no retry relationship is auto-terminated + And InvokeHTTP's retry relationship is auto-terminated When both instances start up Then in the "server" container at least one file with the content "Ut enim ad minim veniam" is placed in the "/tmp/output" directory in less than 10s @@ -115,6 +127,7 @@ Feature: Transfer data from and to MiNiFi using HTTPS And PutFile is EVENT_DRIVEN in the "server" flow And in the "server" flow the "success" relationship of the ListenHTTP processor is connected to the PutFile And PutFile's success relationship is auto-terminated in the "server" flow + And PutFile's failure relationship is auto-terminated in the "server" flow And a GenerateFlowFile processor with the "Data Format" property set to "Text" And the "Unique FlowFiles" property of the GenerateFlowFile processor is set to "false" @@ -127,6 +140,8 @@ Feature: Transfer data from and to MiNiFi using HTTPS And InvokeHTTP's success relationship is auto-terminated And InvokeHTTP's response relationship is auto-terminated And InvokeHTTP's failure relationship is auto-terminated + And InvokeHTTP's no retry relationship is auto-terminated + And InvokeHTTP's retry relationship is auto-terminated When both instances start up Then in the "server" container at least one file with the content "quis nostrud exercitation ullamco laboris nisi" is placed in the "/tmp/output" directory in less than 10s @@ -137,6 +152,7 @@ Feature: Transfer data from and to MiNiFi using HTTPS And a PutFile processor with the "Directory" property set to "/tmp/output" in the "server" flow And in the "server" flow the "success" relationship of the ListenHTTP processor is connected to the PutFile And PutFile's success relationship is auto-terminated in the "server" flow + And PutFile's failure relationship is auto-terminated in the "server" flow And a GenerateFlowFile processor with the "Data Format" property set to "Text" And the "Unique FlowFiles" property of the GenerateFlowFile processor is set to "false" @@ -148,6 +164,8 @@ Feature: Transfer data from and to MiNiFi using HTTPS And InvokeHTTP's success relationship is auto-terminated And InvokeHTTP's response relationship is auto-terminated And InvokeHTTP's failure relationship is auto-terminated + And InvokeHTTP's no retry relationship is auto-terminated + And InvokeHTTP's retry relationship is auto-terminated When both instances start up Then in the "server" container no files are placed in the "/tmp/output" directory in 10s of running time @@ -161,6 +179,7 @@ Feature: Transfer data from and to MiNiFi using HTTPS And PutFile is EVENT_DRIVEN in the "server" flow And in the "server" flow the "success" relationship of the ListenHTTP processor is connected to the PutFile And PutFile's success relationship is auto-terminated in the "server" flow + And PutFile's failure relationship is auto-terminated in the "server" flow And a GenerateFlowFile processor with the "Data Format" property set to "Text" And the "Unique FlowFiles" property of the GenerateFlowFile processor is set to "false" @@ -172,6 +191,8 @@ Feature: Transfer data from and to MiNiFi using HTTPS And InvokeHTTP's success relationship is auto-terminated And InvokeHTTP's response relationship is auto-terminated And InvokeHTTP's failure relationship is auto-terminated + And InvokeHTTP's no retry relationship is auto-terminated + And InvokeHTTP's retry relationship is auto-terminated When both instances start up Then in the "server" container no files are placed in the "/tmp/output" directory in 10s of running time diff --git a/extensions/couchbase/tests/features/couchbase.feature b/extensions/couchbase/tests/features/couchbase.feature index 3464f0410a..6dda030d3e 100644 --- a/extensions/couchbase/tests/features/couchbase.feature +++ b/extensions/couchbase/tests/features/couchbase.feature @@ -97,6 +97,10 @@ Feature: Executing Couchbase operations from MiNiFi-C++ And the "success" relationship of the PutCouchbaseKey processor is connected to the GetCouchbaseKey And the "success" relationship of the GetCouchbaseKey processor is connected to the PutFile And the "success" relationship of the PutFile processor is connected to the LogAttribute + And GetCouchbaseKey's failure relationship is auto-terminated + And GetCouchbaseKey's retry relationship is auto-terminated + And GetCouchbaseKey's original relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And LogAttribute's success relationship is auto-terminated When a Couchbase server is started @@ -133,6 +137,10 @@ Feature: Executing Couchbase operations from MiNiFi-C++ And the "success" relationship of the PutCouchbaseKey processor is connected to the GetCouchbaseKey And the "success" relationship of the GetCouchbaseKey processor is connected to the PutFile And the "success" relationship of the PutFile processor is connected to the LogAttribute + And GetCouchbaseKey's failure relationship is auto-terminated + And GetCouchbaseKey's retry relationship is auto-terminated + And GetCouchbaseKey's original relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And LogAttribute's success relationship is auto-terminated When a Couchbase server is started @@ -170,6 +178,10 @@ Feature: Executing Couchbase operations from MiNiFi-C++ And the "success" relationship of the PutCouchbaseKey processor is connected to the GetCouchbaseKey And the "success" relationship of the GetCouchbaseKey processor is connected to the PutFile And the "success" relationship of the PutFile processor is connected to the LogAttribute + And GetCouchbaseKey's failure relationship is auto-terminated + And GetCouchbaseKey's retry relationship is auto-terminated + And GetCouchbaseKey's original relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And LogAttribute's success relationship is auto-terminated When a Couchbase server is started @@ -202,6 +214,9 @@ Feature: Executing Couchbase operations from MiNiFi-C++ And the "retry" relationship of the PutCouchbaseKey processor is connected to the PutCouchbaseKey And the "success" relationship of the PutCouchbaseKey processor is connected to the GetCouchbaseKey And GetCouchbaseKey's failure relationship is auto-terminated + And GetCouchbaseKey's success relationship is auto-terminated + And GetCouchbaseKey's retry relationship is auto-terminated + And GetCouchbaseKey's original relationship is auto-terminated When a Couchbase server is started And the MiNiFi instance starts up @@ -234,6 +249,10 @@ Feature: Executing Couchbase operations from MiNiFi-C++ And the "success" relationship of the PutCouchbaseKey processor is connected to the GetCouchbaseKey And the "success" relationship of the GetCouchbaseKey processor is connected to the PutFile And the "success" relationship of the PutFile processor is connected to the LogAttribute + And GetCouchbaseKey's failure relationship is auto-terminated + And GetCouchbaseKey's retry relationship is auto-terminated + And GetCouchbaseKey's original relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And LogAttribute's success relationship is auto-terminated When a Couchbase server is started @@ -268,6 +287,10 @@ Feature: Executing Couchbase operations from MiNiFi-C++ And the "success" relationship of the PutCouchbaseKey processor is connected to the GetCouchbaseKey And the "success" relationship of the GetCouchbaseKey processor is connected to the PutFile And the "success" relationship of the PutFile processor is connected to the LogAttribute + And GetCouchbaseKey's failure relationship is auto-terminated + And GetCouchbaseKey's retry relationship is auto-terminated + And GetCouchbaseKey's original relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And LogAttribute's success relationship is auto-terminated When a Couchbase server is started diff --git a/extensions/elasticsearch/tests/features/elasticsearch.feature b/extensions/elasticsearch/tests/features/elasticsearch.feature index 5379847d8e..75bae768ba 100644 --- a/extensions/elasticsearch/tests/features/elasticsearch.feature +++ b/extensions/elasticsearch/tests/features/elasticsearch.feature @@ -33,6 +33,9 @@ Feature: Managing documents on Elasticsearch with PostElasticsearch And PutFile is EVENT_DRIVEN And the "success" relationship of the GetFile processor is connected to the PostElasticsearch And the "success" relationship of the PostElasticsearch processor is connected to the PutFile + And PostElasticsearch's error relationship is auto-terminated + And PostElasticsearch's failure relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And PutFile's success relationship is auto-terminated When the MiNiFi instance starts up @@ -61,6 +64,9 @@ Feature: Managing documents on Elasticsearch with PostElasticsearch And PutFile is EVENT_DRIVEN And the "success" relationship of the GetFile processor is connected to the PostElasticsearch And the "success" relationship of the PostElasticsearch processor is connected to the PutFile + And PostElasticsearch's error relationship is auto-terminated + And PostElasticsearch's failure relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And PutFile's success relationship is auto-terminated When the MiNiFi instance starts up @@ -84,6 +90,9 @@ Feature: Managing documents on Elasticsearch with PostElasticsearch And PutFile is EVENT_DRIVEN And the "success" relationship of the GetFile processor is connected to the PostElasticsearch And the "success" relationship of the PostElasticsearch processor is connected to the PutFile + And PostElasticsearch's error relationship is auto-terminated + And PostElasticsearch's failure relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And PutFile's success relationship is auto-terminated When the MiNiFi instance starts up diff --git a/extensions/elasticsearch/tests/features/opensearch.feature b/extensions/elasticsearch/tests/features/opensearch.feature index 18c949a79e..a26dbce825 100644 --- a/extensions/elasticsearch/tests/features/opensearch.feature +++ b/extensions/elasticsearch/tests/features/opensearch.feature @@ -33,6 +33,9 @@ Feature: PostElasticsearch works on Opensearch (Opensearch doesnt support API Ke And PutFile is EVENT_DRIVEN And the "success" relationship of the GetFile processor is connected to the PostElasticsearch And the "success" relationship of the PostElasticsearch processor is connected to the PutFile + And PostElasticsearch's error relationship is auto-terminated + And PostElasticsearch's failure relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And PutFile's success relationship is auto-terminated When the MiNiFi instance starts up @@ -61,6 +64,9 @@ Feature: PostElasticsearch works on Opensearch (Opensearch doesnt support API Ke And PutFile is EVENT_DRIVEN And the "success" relationship of the GetFile processor is connected to the PostElasticsearch And the "success" relationship of the PostElasticsearch processor is connected to the PutFile + And PostElasticsearch's error relationship is auto-terminated + And PostElasticsearch's failure relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And PutFile's success relationship is auto-terminated When the MiNiFi instance starts up @@ -84,6 +90,9 @@ Feature: PostElasticsearch works on Opensearch (Opensearch doesnt support API Ke And PutFile is EVENT_DRIVEN And the "success" relationship of the GetFile processor is connected to the PostElasticsearch And the "success" relationship of the PostElasticsearch processor is connected to the PutFile + And PostElasticsearch's error relationship is auto-terminated + And PostElasticsearch's failure relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And PutFile's success relationship is auto-terminated When the MiNiFi instance starts up diff --git a/extensions/gcp/tests/features/google_cloud_storage.feature b/extensions/gcp/tests/features/google_cloud_storage.feature index dae15f0bf8..d3ba21458d 100644 --- a/extensions/gcp/tests/features/google_cloud_storage.feature +++ b/extensions/gcp/tests/features/google_cloud_storage.feature @@ -34,6 +34,7 @@ Feature: Sending data to Google Cloud Storage using PutGCSObject And the "success" relationship of the PutGCSObject processor is connected to the PutFile And the "failure" relationship of the PutGCSObject processor is connected to the PutGCSObject And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When all instances start up @@ -95,7 +96,9 @@ Feature: Sending data to Google Cloud Storage using PutGCSObject And PutFile is EVENT_DRIVEN And the "success" relationship of the ListGCSBucket processor is connected to the FetchGCSObject And the "success" relationship of the FetchGCSObject processor is connected to the PutFile + And FetchGCSObject's failure relationship is auto-terminated And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When the MiNiFi instance starts up @@ -124,6 +127,7 @@ Feature: Sending data to Google Cloud Storage using PutGCSObject And a ProxyConfigurationService controller service is set up with proxy configuration And the "success" relationship of the ListGCSBucket processor is connected to the FetchGCSObject And the "success" relationship of the FetchGCSObject processor is connected to the PutFile + And FetchGCSObject's failure relationship is auto-terminated And PutFile's success relationship is auto-terminated And PutFile's failure relationship is auto-terminated @@ -155,7 +159,10 @@ Feature: Sending data to Google Cloud Storage using PutGCSObject And a PutFile processor with the "Directory" property set to "/tmp/output" And the "success" relationship of the ListGCSBucket processor is connected to the DeleteGCSObject And the "success" relationship of the DeleteGCSObject processor is connected to the PutFile + And DeleteGCSObject's success relationship is auto-terminated + And DeleteGCSObject's failure relationship is auto-terminated And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When the MiNiFi instance starts up @@ -184,6 +191,7 @@ Feature: Sending data to Google Cloud Storage using PutGCSObject And a ProxyConfigurationService controller service is set up with proxy configuration And the "success" relationship of the ListGCSBucket processor is connected to the DeleteGCSObject And the "success" relationship of the DeleteGCSObject processor is connected to the PutFile + And DeleteGCSObject's failure relationship is auto-terminated And PutFile's success relationship is auto-terminated And PutFile's failure relationship is auto-terminated diff --git a/extensions/grafana-loki/tests/features/grafana_loki.feature b/extensions/grafana-loki/tests/features/grafana_loki.feature index d23bbf212c..3f0a8ba7c0 100644 --- a/extensions/grafana-loki/tests/features/grafana_loki.feature +++ b/extensions/grafana-loki/tests/features/grafana_loki.feature @@ -25,6 +25,7 @@ Feature: MiNiFi can publish logs to Grafana Loki server And the "Stream Labels" property of the PushGrafanaLokiREST processor is set to "job=minifi,id=docker-test" And the "success" relationship of the TailFile processor is connected to the PushGrafanaLokiREST And PushGrafanaLokiREST's success relationship is auto-terminated + And PushGrafanaLokiREST's failure relationship is auto-terminated When all instances start up Then "log line 1;log line 2;log line 3" lines are published to the Grafana Loki server in less than 60 seconds @@ -39,6 +40,7 @@ Feature: MiNiFi can publish logs to Grafana Loki server And the "Tenant ID" property of the PushGrafanaLokiREST processor is set to "mytenant" And the "success" relationship of the TailFile processor is connected to the PushGrafanaLokiREST And PushGrafanaLokiREST's success relationship is auto-terminated + And PushGrafanaLokiREST's failure relationship is auto-terminated When all instances start up Then "log line 1;log line 2;log line 3" lines are published to the "mytenant" tenant on the Grafana Loki server in less than 60 seconds @@ -54,6 +56,7 @@ Feature: MiNiFi can publish logs to Grafana Loki server And the "SSL Context Service" property of the PushGrafanaLokiREST processor is set to "SSLContextService" And an ssl context service is set up And PushGrafanaLokiREST's success relationship is auto-terminated + And PushGrafanaLokiREST's failure relationship is auto-terminated When all instances start up Then "log line 1;log line 2;log line 3" lines are published using SSL to the Grafana Loki server in less than 60 seconds @@ -70,6 +73,7 @@ Feature: MiNiFi can publish logs to Grafana Loki server And the "Password" property of the PushGrafanaLokiREST processor is set to "password" And the "success" relationship of the TailFile processor is connected to the PushGrafanaLokiREST And PushGrafanaLokiREST's success relationship is auto-terminated + And PushGrafanaLokiREST's failure relationship is auto-terminated When all instances start up Then "log line 1;log line 2;log line 3" lines are published to the Grafana Loki server in less than 60 seconds @@ -83,6 +87,7 @@ Feature: MiNiFi can publish logs to Grafana Loki server And the "Stream Labels" property of the PushGrafanaLokiGrpc processor is set to "job=minifi,id=docker-test" And the "success" relationship of the TailFile processor is connected to the PushGrafanaLokiGrpc And PushGrafanaLokiGrpc's success relationship is auto-terminated + And PushGrafanaLokiGrpc's failure relationship is auto-terminated When all instances start up Then "log line 1;log line 2;log line 3" lines are published to the Grafana Loki server in less than 60 seconds @@ -97,6 +102,7 @@ Feature: MiNiFi can publish logs to Grafana Loki server And the "Tenant ID" property of the PushGrafanaLokiGrpc processor is set to "mytenant" And the "success" relationship of the TailFile processor is connected to the PushGrafanaLokiGrpc And PushGrafanaLokiGrpc's success relationship is auto-terminated + And PushGrafanaLokiGrpc's failure relationship is auto-terminated When all instances start up Then "log line 1;log line 2;log line 3" lines are published to the "mytenant" tenant on the Grafana Loki server in less than 60 seconds @@ -111,6 +117,7 @@ Feature: MiNiFi can publish logs to Grafana Loki server And the "SSL Context Service" property of the PushGrafanaLokiGrpc processor is set to "SSLContextService" And the "success" relationship of the TailFile processor is connected to the PushGrafanaLokiGrpc And PushGrafanaLokiGrpc's success relationship is auto-terminated + And PushGrafanaLokiGrpc's failure relationship is auto-terminated And an ssl context service is set up When all instances start up diff --git a/extensions/kafka/tests/features/consumekafka.feature b/extensions/kafka/tests/features/consumekafka.feature index b23133cd28..ab2bc8359c 100644 --- a/extensions/kafka/tests/features/consumekafka.feature +++ b/extensions/kafka/tests/features/consumekafka.feature @@ -29,6 +29,7 @@ Feature: Receiving data from using Kafka streaming platform using ConsumeKafka And PutFile is EVENT_DRIVEN And the "success" relationship of the ConsumeKafka processor is connected to the PutFile And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And the Kafka server is started And the topic "ConsumeKafkaTest" is initialized on the kafka broker @@ -62,7 +63,10 @@ Feature: Receiving data from using Kafka streaming platform using ConsumeKafka And the "success" relationship of the ConsumeKafka processor is connected to the LogAttribute And the "success" relationship of the LogAttribute processor is connected to the RouteOnAttribute And the "success" relationship of the RouteOnAttribute processor is connected to the PutFile + And RouteOnAttribute's failure relationship is auto-terminated + And RouteOnAttribute's unmatched relationship is auto-terminated And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When all instances start up And a message with content "" is published to the "ConsumeKafkaTest" topic with key "consume_kafka_test_key" @@ -85,6 +89,7 @@ Feature: Receiving data from using Kafka streaming platform using ConsumeKafka And PutFile is EVENT_DRIVEN And the "success" relationship of the ConsumeKafka processor is connected to the PutFile And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When all instances start up And the publisher performs a transaction publishing to the "ConsumeKafkaTest" topic these messages: @@ -116,7 +121,10 @@ Feature: Receiving data from using Kafka streaming platform using ConsumeKafka And the "success" relationship of the ConsumeKafka processor is connected to the LogAttribute And the "success" relationship of the LogAttribute processor is connected to the RouteOnAttribute And the "success" relationship of the RouteOnAttribute processor is connected to the PutFile + And RouteOnAttribute's failure relationship is auto-terminated + And RouteOnAttribute's unmatched relationship is auto-terminated And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When all instances start up And a message with content "" is published to the "ConsumeKafkaTest" topic with headers "" @@ -142,6 +150,7 @@ Feature: Receiving data from using Kafka streaming platform using ConsumeKafka And the "success" relationship of the ConsumeKafka processor is connected to the PutFile And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And the Kafka server is started And the topic "ConsumeKafkaTest" is initialized on the kafka broker @@ -167,6 +176,7 @@ Feature: Receiving data from using Kafka streaming platform using ConsumeKafka And the "success" relationship of the ConsumeKafka processor is connected to the LogAttribute And the "success" relationship of the LogAttribute processor is connected to the PutFile And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When all instances start up And 1000 kafka messages are sent to the topic "ConsumeKafkaTest" @@ -190,6 +200,7 @@ Feature: Receiving data from using Kafka streaming platform using ConsumeKafka And an ssl context service is set up for ConsumeKafka And the "success" relationship of the ConsumeKafka processor is connected to the PutFile And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When all instances start up And a message with content "Through the Looking-Glass" is published to the "ConsumeKafkaTest" topic @@ -212,6 +223,7 @@ Feature: Receiving data from using Kafka streaming platform using ConsumeKafka And an ssl context service is set up for ConsumeKafka And the "success" relationship of the ConsumeKafka processor is connected to the PutFile And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When all instances start up And a message with content "Through the Looking-Glass" is published to the "ConsumeKafkaTest" topic @@ -224,6 +236,8 @@ Feature: Receiving data from using Kafka streaming platform using ConsumeKafka And ConsumeKafka processor is set up to communicate with that server And a PutFile processor with the "Directory" property set to "/tmp/output" And the "success" relationship of the ConsumeKafka processor is connected to the PutFile + And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And these processor properties are set | processor name | property name | property value | | ConsumeKafka | Kafka Brokers | kafka-server-${scenario_id}:9094 | @@ -249,6 +263,7 @@ Feature: Receiving data from using Kafka streaming platform using ConsumeKafka And PutFile is EVENT_DRIVEN And the "success" relationship of the ConsumeKafka processor is connected to the PutFile And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And the Kafka server is started And the topic "ConsumeKafkaTest" is initialized on the kafka broker @@ -303,7 +318,9 @@ Feature: Receiving data from using Kafka streaming platform using ConsumeKafka And the "success" relationship of the Consumed processor is connected to the ConsumeKafka And the "committed" relationship of the ConsumeKafka processor is connected to the Committed And Consumed's success relationship is auto-terminated + And Consumed's failure relationship is auto-terminated And Committed's success relationship is auto-terminated + And Committed's failure relationship is auto-terminated And the Kafka server is started And the topic "ConsumeKafkaTest" is initialized on the kafka broker diff --git a/extensions/kafka/tests/features/publishkafka.feature b/extensions/kafka/tests/features/publishkafka.feature index aac5439a3f..8bbf1ba6a1 100644 --- a/extensions/kafka/tests/features/publishkafka.feature +++ b/extensions/kafka/tests/features/publishkafka.feature @@ -49,7 +49,9 @@ Feature: Sending data to using Kafka streaming platform using PublishKafka And the "success" relationship of the UpdateAttribute processor is connected to the PublishKafka And the "success" relationship of the PublishKafka processor is connected to the PutFile And the "failure" relationship of the PublishKafka processor is connected to the PublishKafka + And UpdateAttribute's failure relationship is auto-terminated And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When both instances start up Then a single file with the content "test" is placed in the "/tmp/output" directory in less than 60 seconds @@ -80,7 +82,9 @@ Feature: Sending data to using Kafka streaming platform using PublishKafka And PutFile is EVENT_DRIVEN And the "success" relationship of the GetFile processor is connected to the PublishKafka And the "failure" relationship of the PublishKafka processor is connected to the PutFile + And PublishKafka's success relationship is auto-terminated And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When the MiNiFi instance starts up Then a single file with the content "no broker" is placed in the "/tmp/output" directory in less than 60 seconds @@ -111,6 +115,7 @@ Feature: Sending data to using Kafka streaming platform using PublishKafka And the "success" relationship of the PublishKafka processor is connected to the PutFile And the "failure" relationship of the PublishKafka processor is connected to the PublishKafka And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When both instances start up Then a single file with the content "test" is placed in the "/tmp/output" directory in less than 60 seconds @@ -141,6 +146,7 @@ Feature: Sending data to using Kafka streaming platform using PublishKafka And the "success" relationship of the PublishKafka processor is connected to the PutFile And the "failure" relationship of the PublishKafka processor is connected to the PublishKafka And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When both instances start up Then a single file with the content "test" is placed in the "/tmp/output" directory in less than 60 seconds @@ -174,6 +180,7 @@ Feature: Sending data to using Kafka streaming platform using PublishKafka And the "success" relationship of the PublishKafka processor is connected to the PutFile And the "failure" relationship of the PublishKafka processor is connected to the PublishKafka And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When both instances start up Then a single file with the content "test" is placed in the "/tmp/output" directory in less than 60 seconds @@ -205,6 +212,7 @@ Feature: Sending data to using Kafka streaming platform using PublishKafka And the "success" relationship of the PublishKafka processor is connected to the PutFile And the "failure" relationship of the PublishKafka processor is connected to the PublishKafka And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When both instances start up Then a single file with the content "test" is placed in the "/tmp/output" directory in less than 60 seconds @@ -233,6 +241,7 @@ Feature: Sending data to using Kafka streaming platform using PublishKafka And the "success" relationship of the PublishKafka processor is connected to the PutFile And the "failure" relationship of the PublishKafka processor is connected to the PublishKafka And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When both instances start up Then a single file with the content "test" is placed in the "/tmp/output" directory in less than 60 seconds diff --git a/extensions/kubernetes/tests/features/kubernetes.feature b/extensions/kubernetes/tests/features/kubernetes.feature index 33ccdf2154..1a4f8fab35 100644 --- a/extensions/kubernetes/tests/features/kubernetes.feature +++ b/extensions/kubernetes/tests/features/kubernetes.feature @@ -27,7 +27,8 @@ Feature: MiNiFi can get logs and metrics from Kubernetes pods And the "Directory" property of the PutFile processor is set to "/tmp/output" And the "Conflict Resolution Strategy" property of the PutFile processor is set to "ignore" And the "success" relationship of the TailFile processor is connected to the PutFile - And the "success" relationship of the PutFile processor is auto-terminated + And PutFile's failure relationship is auto-terminated + And PutFile's success relationship is auto-terminated When the MiNiFi instance starts up Then the content of at least one file in the "/tmp/output" directory matches the '[0-9TZ:.-]+ stdout F Hello World!' regex in less than 30 seconds And the content of at least one file in the "/tmp/output" directory matches the '[0-9TZ:.-]+ stdout F Hello again, World!' regex in less than 30 seconds @@ -43,7 +44,8 @@ Feature: MiNiFi can get logs and metrics from Kubernetes pods And the "Directory" property of the PutFile processor is set to "/tmp/output" And the "Conflict Resolution Strategy" property of the PutFile processor is set to "ignore" And the "success" relationship of the TailFile processor is connected to the PutFile - And the "success" relationship of the PutFile processor is auto-terminated + And PutFile's failure relationship is auto-terminated + And PutFile's success relationship is auto-terminated When the MiNiFi instance starts up Then the content of at least one file in the "/tmp/output" directory matches the '[0-9TZ:.-]+ stdout F Hello World!' regex in less than 30 seconds @@ -58,7 +60,8 @@ Feature: MiNiFi can get logs and metrics from Kubernetes pods And the "Directory" property of the PutFile processor is set to "/tmp/output" And the "Conflict Resolution Strategy" property of the PutFile processor is set to "ignore" And the "success" relationship of the TailFile processor is connected to the PutFile - And the "success" relationship of the PutFile processor is auto-terminated + And PutFile's failure relationship is auto-terminated + And PutFile's success relationship is auto-terminated When the MiNiFi instance starts up Then the content of at least one file in the "/tmp/output" directory matches the '[0-9TZ:.-]+ stdout F Hello again, World!' regex in less than 30 seconds @@ -84,7 +87,8 @@ Feature: MiNiFi can get logs and metrics from Kubernetes pods And a PutFile processor in the Kubernetes cluster And the "Directory" property of the PutFile processor is set to "/tmp/output" And the "success" relationship of the CollectKubernetesPodMetrics processor is connected to the PutFile - And the "success" relationship of the PutFile processor is auto-terminated + And PutFile's failure relationship is auto-terminated + And PutFile's success relationship is auto-terminated When the MiNiFi instance starts up Then the content of at least one file in the "/tmp/output" directory matches the '"kind":"PodMetricsList","apiVersion":"metrics.k8s.io/v1beta1"' regex in less than 2 minutes @@ -94,7 +98,8 @@ Feature: MiNiFi can get logs and metrics from Kubernetes pods And a PutFile processor in the Kubernetes cluster And the "Directory" property of the PutFile processor is set to "/tmp/output" And the "success" relationship of the CollectKubernetesPodMetrics processor is connected to the PutFile - And the "success" relationship of the PutFile processor is auto-terminated + And PutFile's failure relationship is auto-terminated + And PutFile's success relationship is auto-terminated When the MiNiFi instance starts up Then the content of at least one file in the "/tmp/output" directory matches the '"metadata":{"name":"hello-world-one","namespace":"default"' regex in less than 2 minutes @@ -104,6 +109,7 @@ Feature: MiNiFi can get logs and metrics from Kubernetes pods And a PutFile processor in the Kubernetes cluster And the "Directory" property of the PutFile processor is set to "/tmp/output" And the "success" relationship of the CollectKubernetesPodMetrics processor is connected to the PutFile - And the "success" relationship of the PutFile processor is auto-terminated + And PutFile's failure relationship is auto-terminated + And PutFile's success relationship is auto-terminated When the MiNiFi instance starts up Then the content of at least one file in the "/tmp/output" directory matches the '"containers":\[{"name":"echo-two","usage":{"cpu":"0","memory":' regex in less than 2 minutes diff --git a/extensions/llamacpp/tests/features/llamacpp.feature b/extensions/llamacpp/tests/features/llamacpp.feature index 5ff088db36..e0a32712a5 100644 --- a/extensions/llamacpp/tests/features/llamacpp.feature +++ b/extensions/llamacpp/tests/features/llamacpp.feature @@ -24,6 +24,8 @@ Feature: Run language model inference using LlamaCpp processor And a LogAttribute processor with the "Log Payload" property set to "true" And the "success" relationship of the GenerateFlowFile processor is connected to the RunLlamaCppInference And the "success" relationship of the RunLlamaCppInference processor is connected to the LogAttribute + And RunLlamaCppInference's failure relationship is auto-terminated + And LogAttribute's success relationship is auto-terminated When all instances start up Then the Minifi logs contain the following message: "banana" in less than 60 seconds @@ -38,7 +40,9 @@ Feature: Run language model inference using LlamaCpp processor And a PutFile processor with the "Directory" property set to "/tmp/output" And the "success" relationship of the GetFile processor is connected to the RunLlamaCppInference And the "success" relationship of the RunLlamaCppInference processor is connected to the PutFile + And RunLlamaCppInference's failure relationship is auto-terminated + And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When all instances start up Then a single file with the content "minifi" is placed in the "/tmp/output" directory in less than 600 seconds - diff --git a/extensions/lua/tests/features/lua_script.feature b/extensions/lua/tests/features/lua_script.feature index c4c57ba0aa..5e81b2cb93 100644 --- a/extensions/lua/tests/features/lua_script.feature +++ b/extensions/lua/tests/features/lua_script.feature @@ -26,6 +26,8 @@ Feature: MiNiFi can execute Lua scripts And the max concurrent tasks attribute of the ExecuteScript processor is set to 3 And the "success" relationship of the GenerateFlowFile processor is connected to the ExecuteScript And ExecuteScript's success relationship is auto-terminated + And ExecuteScript's failure relationship is auto-terminated + And ExecuteScript's original relationship is auto-terminated When the MiNiFi instance starts up Then the Minifi logs contain the following message: "Sleeping forever" 3 times after 5 seconds diff --git a/extensions/mqtt/tests/features/mqtt.feature b/extensions/mqtt/tests/features/mqtt.feature index f77d5e4ea9..a277c18e9b 100644 --- a/extensions/mqtt/tests/features/mqtt.feature +++ b/extensions/mqtt/tests/features/mqtt.feature @@ -28,7 +28,9 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And PutFile is EVENT_DRIVEN And the "success" relationship of the GetFile processor is connected to the PublishMQTT And the "success" relationship of the PublishMQTT processor is connected to the PutFile + And PublishMQTT's failure relationship is auto-terminated And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And an MQTT broker is started @@ -52,6 +54,7 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And the "success" relationship of the PublishMQTT processor is connected to the PutFile And the "failure" relationship of the PublishMQTT processor is connected to the PutFile And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When the MiNiFi instance starts up Then no files are placed in the "/tmp/output" directory in 30 seconds of running time @@ -70,7 +73,9 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And PutFile is EVENT_DRIVEN And the "success" relationship of the GetFile processor is connected to the PublishMQTT And the "success" relationship of the PublishMQTT processor is connected to the PutFile + And PublishMQTT's failure relationship is auto-terminated And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When the MiNiFi instance starts up Then no files are placed in the "/tmp/output" directory in 10 seconds of running time @@ -91,6 +96,7 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And the "MQTT Version" property of the PublishMQTT processor is set to "" And the "success" relationship of the GetFile processor is connected to the PublishMQTT And PublishMQTT's success relationship is auto-terminated + And PublishMQTT's failure relationship is auto-terminated And a ConsumeMQTT processor set up to communicate with an MQTT broker instance And the "MQTT Version" property of the ConsumeMQTT processor is set to "" @@ -101,6 +107,7 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And the "success" relationship of the ConsumeMQTT processor is connected to the PutFile And the "success" relationship of the PutFile processor is connected to the LogAttribute And LogAttribute's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And an MQTT broker is started @@ -132,6 +139,7 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And the "MQTT Version" property of the PublishMQTT processor is set to "" And the "success" relationship of the GetFile processor is connected to the PublishMQTT And PublishMQTT's success relationship is auto-terminated + And PublishMQTT's failure relationship is auto-terminated And a ConsumeMQTT processor set up to communicate with an MQTT broker instance And the "Topic" property of the ConsumeMQTT processor is set to "" @@ -140,6 +148,7 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And PutFile is EVENT_DRIVEN And the "success" relationship of the ConsumeMQTT processor is connected to the PutFile And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And an MQTT broker is started @@ -166,6 +175,7 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And the "Quality of Service" property of the PublishMQTT processor is set to "1" in the "publisher-client" flow And in the "publisher-client" flow the "success" relationship of the GetFile processor is connected to the PublishMQTT And PublishMQTT's success relationship is auto-terminated in the "publisher-client" flow + And PublishMQTT's failure relationship is auto-terminated in the "publisher-client" flow # consuming MQTT client And a ConsumeMQTT processor set up to communicate with an MQTT broker instance in the "consumer-client" flow @@ -177,6 +187,7 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And PutFile is EVENT_DRIVEN in the "consumer-client" flow And in the "consumer-client" flow the "success" relationship of the ConsumeMQTT processor is connected to the PutFile And PutFile's success relationship is auto-terminated in the "consumer-client" flow + And PutFile's failure relationship is auto-terminated in the "consumer-client" flow And an MQTT broker is started @@ -205,6 +216,7 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And the "Topic" property of the PublishMQTT processor is set to "" And the "success" relationship of the GetFile processor is connected to the PublishMQTT And PublishMQTT's success relationship is auto-terminated + And PublishMQTT's failure relationship is auto-terminated And a ConsumeMQTT processor set up to communicate with an MQTT broker instance And the "MQTT Version" property of the ConsumeMQTT processor is set to "" @@ -213,6 +225,7 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And PutFile is EVENT_DRIVEN And the "success" relationship of the ConsumeMQTT processor is connected to the PutFile And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And an MQTT broker is started @@ -239,6 +252,7 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And the "Quality of Service" property of the PublishMQTT processor is set to "0" And the "success" relationship of the GetFile processor is connected to the PublishMQTT And PublishMQTT's success relationship is auto-terminated + And PublishMQTT's failure relationship is auto-terminated And a ConsumeMQTT processor set up to communicate with an MQTT broker instance And the "MQTT Version" property of the ConsumeMQTT processor is set to "" @@ -247,6 +261,7 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And PutFile is EVENT_DRIVEN And the "success" relationship of the ConsumeMQTT processor is connected to the PutFile And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And an MQTT broker is started @@ -271,6 +286,7 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And the "Quality of Service" property of the PublishMQTT processor is set to "1" And the "success" relationship of the GetFile processor is connected to the PublishMQTT And PublishMQTT's success relationship is auto-terminated + And PublishMQTT's failure relationship is auto-terminated And a ConsumeMQTT processor set up to communicate with an MQTT broker instance And the "MQTT Version" property of the ConsumeMQTT processor is set to "" @@ -279,6 +295,7 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And PutFile is EVENT_DRIVEN And the "success" relationship of the ConsumeMQTT processor is connected to the PutFile And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And an MQTT broker is started @@ -305,6 +322,7 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And the "Quality of Service" property of the PublishMQTT processor is set to "2" And the "success" relationship of the GetFile processor is connected to the PublishMQTT And PublishMQTT's success relationship is auto-terminated + And PublishMQTT's failure relationship is auto-terminated And a ConsumeMQTT processor set up to communicate with an MQTT broker instance And the "MQTT Version" property of the ConsumeMQTT processor is set to "" @@ -313,6 +331,7 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And PutFile is EVENT_DRIVEN And the "success" relationship of the ConsumeMQTT processor is connected to the PutFile And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And an MQTT broker is started @@ -346,6 +365,7 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And the "Retain" property of the PublishMQTT processor is set to "true" in the "publisher-client" flow And in the "publisher-client" flow the "success" relationship of the GetFile processor is connected to the PublishMQTT And PublishMQTT's success relationship is auto-terminated in the "publisher-client" flow + And PublishMQTT's failure relationship is auto-terminated in the "publisher-client" flow # consuming MQTT client And a ConsumeMQTT processor set up to communicate with an MQTT broker instance in the "consumer-client" flow @@ -354,6 +374,7 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And PutFile is EVENT_DRIVEN in the "consumer-client" flow And in the "consumer-client" flow the "success" relationship of the ConsumeMQTT processor is connected to the PutFile And PutFile's success relationship is auto-terminated in the "consumer-client" flow + And PutFile's failure relationship is auto-terminated in the "consumer-client" flow And an MQTT broker is started @@ -381,6 +402,7 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And the "Last Will Message" property of the PublishMQTT processor is set to "last_will_message" in the "publisher-client" flow And in the "publisher-client" flow the "success" relationship of the GetFile processor is connected to the PublishMQTT And PublishMQTT's success relationship is auto-terminated in the "publisher-client" flow + And PublishMQTT's failure relationship is auto-terminated in the "publisher-client" flow # consuming MQTT client set to consume last will topic And a ConsumeMQTT processor set up to communicate with an MQTT broker instance in the "consumer-client" flow @@ -390,6 +412,7 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And PutFile is EVENT_DRIVEN in the "consumer-client" flow And in the "consumer-client" flow the "success" relationship of the ConsumeMQTT processor is connected to the PutFile And PutFile's success relationship is auto-terminated in the "consumer-client" flow + And PutFile's failure relationship is auto-terminated in the "consumer-client" flow And an MQTT broker is started @@ -409,6 +432,7 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT Given a ConsumeMQTT processor set up to communicate with an MQTT broker instance And the "MQTT Version" property of the ConsumeMQTT processor is set to "" And the "Keep Alive Interval" property of the ConsumeMQTT processor is set to "1 sec" + And ConsumeMQTT's success relationship is auto-terminated And an MQTT broker is started @@ -431,6 +455,7 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And the "Quality of Service" property of the PublishMQTT processor is set to "1" in the "publisher-client" flow And in the "publisher-client" flow the "success" relationship of the GetFile processor is connected to the PublishMQTT And PublishMQTT's success relationship is auto-terminated in the "publisher-client" flow + And PublishMQTT's failure relationship is auto-terminated in the "publisher-client" flow # consuming MQTT client And a ConsumeMQTT processor set up to communicate with an MQTT broker instance in the "consumer-client" flow @@ -442,6 +467,7 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And PutFile is EVENT_DRIVEN in the "consumer-client" flow And in the "consumer-client" flow the "success" relationship of the ConsumeMQTT processor is connected to the PutFile And PutFile's success relationship is auto-terminated in the "consumer-client" flow + And PutFile's failure relationship is auto-terminated in the "consumer-client" flow And an MQTT broker is started @@ -474,7 +500,9 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And the "MQTT Version" property of the PublishMQTT processor is set to "5.0" in the "publisher-client" flow And in the "publisher-client" flow the "success" relationship of the GetFile processor is connected to the UpdateAttribute And in the "publisher-client" flow the "success" relationship of the UpdateAttribute processor is connected to the PublishMQTT + And UpdateAttribute's failure relationship is auto-terminated in the "publisher-client" flow And PublishMQTT's success relationship is auto-terminated in the "publisher-client" flow + And PublishMQTT's failure relationship is auto-terminated in the "publisher-client" flow # consuming MQTT client: ConsumeMQTT -> RouteOnAttribute (my_attr1) -> RouteOnAttribute (my_attr2) -> PutFile And a ConsumeMQTT processor set up to communicate with an MQTT broker instance in the "consumer-client" flow @@ -490,7 +518,12 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And in the "consumer-client" flow the "success" relationship of the ConsumeMQTT processor is connected to the RouteAttr1 And in the "consumer-client" flow the "matched_my_attr1" relationship of the RouteAttr1 processor is connected to the RouteAttr2 And in the "consumer-client" flow the "matched_my_attr2" relationship of the RouteAttr2 processor is connected to the PutFile + And RouteAttr1's failure relationship is auto-terminated in the "consumer-client" flow + And RouteAttr1's unmatched relationship is auto-terminated in the "consumer-client" flow + And RouteAttr2's failure relationship is auto-terminated in the "consumer-client" flow + And RouteAttr2's unmatched relationship is auto-terminated in the "consumer-client" flow And PutFile's success relationship is auto-terminated in the "consumer-client" flow + And PutFile's failure relationship is auto-terminated in the "consumer-client" flow And an MQTT broker is started @@ -510,6 +543,7 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And the "Content Type" property of the PublishMQTT processor is set to "text/plain" in the "publisher-client" flow And in the "publisher-client" flow the "success" relationship of the GetFile processor is connected to the PublishMQTT And PublishMQTT's success relationship is auto-terminated in the "publisher-client" flow + And PublishMQTT's failure relationship is auto-terminated in the "publisher-client" flow # consuming MQTT client: ConsumeMQTT (Attribute From Content Type: content_type) -> RouteOnAttribute (content_type = text/plain) -> PutFile And a ConsumeMQTT processor set up to communicate with an MQTT broker instance in the "consumer-client" flow @@ -522,7 +556,10 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And PutFile is EVENT_DRIVEN in the "consumer-client" flow And in the "consumer-client" flow the "success" relationship of the ConsumeMQTT processor is connected to the RouteOnAttribute And in the "consumer-client" flow the "matched_content_type" relationship of the RouteOnAttribute processor is connected to the PutFile + And RouteOnAttribute's failure relationship is auto-terminated in the "consumer-client" flow + And RouteOnAttribute's unmatched relationship is auto-terminated in the "consumer-client" flow And PutFile's success relationship is auto-terminated in the "consumer-client" flow + And PutFile's failure relationship is auto-terminated in the "consumer-client" flow And an MQTT broker is started @@ -544,6 +581,7 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And the "Last Will Content Type" property of the PublishMQTT processor is set to "text/plain" in the "publisher-client" flow And in the "publisher-client" flow the "success" relationship of the GetFile processor is connected to the PublishMQTT And PublishMQTT's success relationship is auto-terminated in the "publisher-client" flow + And PublishMQTT's failure relationship is auto-terminated in the "publisher-client" flow # consuming MQTT client: ConsumeMQTT (Attribute From Content Type: content_type, Topic: last_will_topic) -> RouteOnAttribute (content_type = text/plain) -> PutFile And a ConsumeMQTT processor set up to communicate with an MQTT broker instance in the "consumer-client" flow @@ -557,7 +595,10 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And PutFile is EVENT_DRIVEN in the "consumer-client" flow And in the "consumer-client" flow the "success" relationship of the ConsumeMQTT processor is connected to the RouteOnAttribute And in the "consumer-client" flow the "matched_content_type" relationship of the RouteOnAttribute processor is connected to the PutFile + And RouteOnAttribute's failure relationship is auto-terminated in the "consumer-client" flow + And RouteOnAttribute's unmatched relationship is auto-terminated in the "consumer-client" flow And PutFile's success relationship is auto-terminated in the "consumer-client" flow + And PutFile's failure relationship is auto-terminated in the "consumer-client" flow And an MQTT broker is started @@ -583,6 +624,8 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And the "success" relationship of the ConsumeMQTT processor is connected to the PutFile And the "success" relationship of the PutFile processor is connected to the LogAttribute And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated + And LogAttribute's success relationship is auto-terminated And an MQTT broker is started @@ -613,7 +656,10 @@ Feature: Sending data to MQTT streaming platform using PublishMQTT And the "success" relationship of the GetFile processor is connected to the PublishMQTT And the "success" relationship of the PublishMQTT processor is connected to the UpdateAttribute And the "success" relationship of the UpdateAttribute processor is connected to the PutFile + And PublishMQTT's failure relationship is auto-terminated + And UpdateAttribute's failure relationship is auto-terminated And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And an MQTT broker is started diff --git a/extensions/opc/tests/features/opcua.feature b/extensions/opc/tests/features/opcua.feature index bb8036152b..4e7f8f6466 100644 --- a/extensions/opc/tests/features/opcua.feature +++ b/extensions/opc/tests/features/opcua.feature @@ -26,7 +26,7 @@ Feature: Putting and fetching data to OPC UA server And PutOPCProcessor is EVENT_DRIVEN in the "create-opc-ua-node" flow And a FetchOPCProcessor processor in the "fetch-opc-ua-node" flow And a PutFile processor with the "Directory" property set to "/tmp/output" in the "fetch-opc-ua-node" flow - And PutFile's success relationship is auto-terminated in the "fetch-opc-ua-node" flow + And PutFile is EVENT_DRIVEN in the "fetch-opc-ua-node" flow And these processor properties are set in the "create-opc-ua-node" flow | processor name | property name | property value | @@ -48,6 +48,11 @@ Feature: Putting and fetching data to OPC UA server And in the "create-opc-ua-node" flow the "success" relationship of the GetFile processor is connected to the PutOPCProcessor And in the "fetch-opc-ua-node" flow the "success" relationship of the FetchOPCProcessor processor is connected to the PutFile + And PutOPCProcessor's success relationship is auto-terminated in the "create-opc-ua-node" flow + And PutOPCProcessor's failure relationship is auto-terminated in the "create-opc-ua-node" flow + And FetchOPCProcessor's failure relationship is auto-terminated in the "fetch-opc-ua-node" flow + And PutFile's success relationship is auto-terminated in the "fetch-opc-ua-node" flow + And PutFile's failure relationship is auto-terminated in the "fetch-opc-ua-node" flow And an OPC UA server is set up @@ -71,10 +76,9 @@ Feature: Putting and fetching data to OPC UA server And a directory at "/tmp/input" has a file with the content "" in the "update-opc-ua-node" flow And a PutOPCProcessor processor in the "update-opc-ua-node" flow And PutOPCProcessor is EVENT_DRIVEN in the "update-opc-ua-node" flow - And PutOPCProcessor's success relationship is auto-terminated in the "update-opc-ua-node" flow And a FetchOPCProcessor processor in the "fetch-opc-ua-node" flow And a PutFile processor with the "Directory" property set to "/tmp/output" in the "fetch-opc-ua-node" flow - And PutFile's success relationship is auto-terminated in the "fetch-opc-ua-node" flow + And PutFile is EVENT_DRIVEN in the "fetch-opc-ua-node" flow And these processor properties are set in the "update-opc-ua-node" flow | processor name | property name | property value | @@ -96,6 +100,11 @@ Feature: Putting and fetching data to OPC UA server And in the "update-opc-ua-node" flow the "success" relationship of the GetFile processor is connected to the PutOPCProcessor And in the "fetch-opc-ua-node" flow the "success" relationship of the FetchOPCProcessor processor is connected to the PutFile + And PutOPCProcessor's success relationship is auto-terminated in the "update-opc-ua-node" flow + And PutOPCProcessor's failure relationship is auto-terminated in the "update-opc-ua-node" flow + And FetchOPCProcessor's failure relationship is auto-terminated in the "fetch-opc-ua-node" flow + And PutFile's success relationship is auto-terminated in the "fetch-opc-ua-node" flow + And PutFile's failure relationship is auto-terminated in the "fetch-opc-ua-node" flow And an OPC UA server is set up @@ -124,7 +133,6 @@ Feature: Putting and fetching data to OPC UA server And PutOPCProcessor is EVENT_DRIVEN in the "create-opc-ua-node" flow And a FetchOPCProcessor processor in the "fetch-opc-ua-node" flow And a PutFile processor with the "Directory" property set to "/tmp/output" in the "fetch-opc-ua-node" flow - And PutFile's success relationship is auto-terminated in the "fetch-opc-ua-node" flow And PutFile is EVENT_DRIVEN in the "fetch-opc-ua-node" flow And the OPC UA server certificate files are placed in the "/tmp/resources/opcua/" directory in the MiNiFi container "create-opc-ua-node" And the OPC UA server certificate files are placed in the "/tmp/resources/opcua/" directory in the MiNiFi container "fetch-opc-ua-node" @@ -156,6 +164,11 @@ Feature: Putting and fetching data to OPC UA server And in the "create-opc-ua-node" flow the "success" relationship of the GetFile processor is connected to the PutOPCProcessor And in the "fetch-opc-ua-node" flow the "success" relationship of the FetchOPCProcessor processor is connected to the PutFile + And PutOPCProcessor's success relationship is auto-terminated in the "create-opc-ua-node" flow + And PutOPCProcessor's failure relationship is auto-terminated in the "create-opc-ua-node" flow + And FetchOPCProcessor's failure relationship is auto-terminated in the "fetch-opc-ua-node" flow + And PutFile's success relationship is auto-terminated in the "fetch-opc-ua-node" flow + And PutFile's failure relationship is auto-terminated in the "fetch-opc-ua-node" flow And an OPC UA server is set up @@ -171,7 +184,6 @@ Feature: Putting and fetching data to OPC UA server And PutOPCProcessor is EVENT_DRIVEN in the "create-opc-ua-node" flow And a FetchOPCProcessor processor in the "fetch-opc-ua-node" flow And a PutFile processor with the "Directory" property set to "/tmp/output" in the "fetch-opc-ua-node" flow - And PutFile's success relationship is auto-terminated in the "fetch-opc-ua-node" flow And PutFile is EVENT_DRIVEN in the "fetch-opc-ua-node" flow And these processor properties are set in the "create-opc-ua-node" flow | processor name | property name | property value | @@ -197,6 +209,11 @@ Feature: Putting and fetching data to OPC UA server And in the "create-opc-ua-node" flow the "success" relationship of the GetFile processor is connected to the PutOPCProcessor And in the "fetch-opc-ua-node" flow the "success" relationship of the FetchOPCProcessor processor is connected to the PutFile + And PutOPCProcessor's success relationship is auto-terminated in the "create-opc-ua-node" flow + And PutOPCProcessor's failure relationship is auto-terminated in the "create-opc-ua-node" flow + And FetchOPCProcessor's failure relationship is auto-terminated in the "fetch-opc-ua-node" flow + And PutFile's success relationship is auto-terminated in the "fetch-opc-ua-node" flow + And PutFile's failure relationship is auto-terminated in the "fetch-opc-ua-node" flow And an OPC UA server is set up with access control @@ -212,7 +229,6 @@ Feature: Putting and fetching data to OPC UA server And PutOPCProcessor is EVENT_DRIVEN in the "create-opc-ua-node" flow And a FetchOPCProcessor processor in the "fetch-opc-ua-node" flow And a PutFile processor with the "Directory" property set to "/tmp/output" in the "fetch-opc-ua-node" flow - And PutFile's success relationship is auto-terminated in the "fetch-opc-ua-node" flow And PutFile is EVENT_DRIVEN in the "fetch-opc-ua-node" flow And the OPC UA server certificate files are placed in the "/tmp/resources/opcua/" directory in the MiNiFi container "create-opc-ua-node" And the OPC UA server certificate files are placed in the "/tmp/resources/opcua/" directory in the MiNiFi container "fetch-opc-ua-node" @@ -248,6 +264,11 @@ Feature: Putting and fetching data to OPC UA server And in the "create-opc-ua-node" flow the "success" relationship of the GetFile processor is connected to the PutOPCProcessor And in the "fetch-opc-ua-node" flow the "success" relationship of the FetchOPCProcessor processor is connected to the PutFile + And PutOPCProcessor's success relationship is auto-terminated in the "create-opc-ua-node" flow + And PutOPCProcessor's failure relationship is auto-terminated in the "create-opc-ua-node" flow + And FetchOPCProcessor's failure relationship is auto-terminated in the "fetch-opc-ua-node" flow + And PutFile's success relationship is auto-terminated in the "fetch-opc-ua-node" flow + And PutFile's failure relationship is auto-terminated in the "fetch-opc-ua-node" flow And an OPC UA server is set up diff --git a/extensions/prometheus/tests/features/prometheus.feature b/extensions/prometheus/tests/features/prometheus.feature index 02b780be9e..432eb9b711 100644 --- a/extensions/prometheus/tests/features/prometheus.feature +++ b/extensions/prometheus/tests/features/prometheus.feature @@ -23,6 +23,7 @@ Feature: MiNiFi can publish metrics to Prometheus server And PutFile is EVENT_DRIVEN And the "success" relationship of the GetFile1 processor is connected to the PutFile And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And Prometheus is enabled in MiNiFi And a Prometheus server is set up When all instances start up @@ -44,6 +45,7 @@ Feature: MiNiFi can publish metrics to Prometheus server And PutFile is EVENT_DRIVEN And the "success" relationship of the GetFile1 processor is connected to the PutFile And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And Prometheus with SSL is enabled in MiNiFi And a Prometheus server is set up with SSL When all instances start up diff --git a/extensions/python/ExecutePythonProcessor.cpp b/extensions/python/ExecutePythonProcessor.cpp index 7ef4ae73a4..077a3773ec 100644 --- a/extensions/python/ExecutePythonProcessor.cpp +++ b/extensions/python/ExecutePythonProcessor.cpp @@ -41,12 +41,12 @@ void ExecutePythonProcessor::initialize() { // only include supported properties that are defined in the python processor setSupportedProperties(std::span(python_properties_)); setSupportedRelationships(Relationships); + addAutoTerminatedRelationship(Original); logger_->log_debug("Processor has been initialized."); } void ExecutePythonProcessor::onSchedule(core::ProcessContext& context, core::ProcessSessionFactory& /*sessionFactory*/) { gsl_Expects(python_script_engine_); - context.addAutoTerminatedRelationship(Original); reloadScriptFile(); python_script_engine_->eval(script_to_exec_); python_script_engine_->onSchedule(context); diff --git a/extensions/python/PythonCreator.h b/extensions/python/PythonCreator.h index fa7c5f0283..2d6cba6ccd 100644 --- a/extensions/python/PythonCreator.h +++ b/extensions/python/PythonCreator.h @@ -52,6 +52,7 @@ class DummyProcessorDescriptor : public core::ProcessorDescriptor { void setSupportedRelationships(std::span /*relationships*/) override {} void setSupportedProperties(std::span /*properties*/) override {} void setSupportedProperties(std::span /*properties*/) override {} + void addAutoTerminatedRelationship(const core::RelationshipDefinition& /*relationship*/) override {} }; class DummyLogger : public core::logging::AdvancedLogger { diff --git a/extensions/python/tests/ExecutePythonProcessorTests.cpp b/extensions/python/tests/ExecutePythonProcessorTests.cpp index b1c7b8809f..cdc937c337 100644 --- a/extensions/python/tests/ExecutePythonProcessorTests.cpp +++ b/extensions/python/tests/ExecutePythonProcessorTests.cpp @@ -202,4 +202,10 @@ TEST_CASE_METHOD(SimplePythonFlowFileTransferTest, "Stateful execution", "[execu testsStatefulProcessor(); } +TEST_CASE_METHOD(SimplePythonFlowFileTransferTest, "Original relationship is auto-terminated on initialization", "[executePythonProcessorOriginal]") { + auto* processor = addExecutePythonProcessorToPlan("passthrough_processor_transfering_to_success.py", false); + REQUIRE(processor); + CHECK(processor->isAutoTerminated(minifi::extensions::python::processors::ExecutePythonProcessor::Original)); +} + } // namespace diff --git a/extensions/python/tests/features/python.feature b/extensions/python/tests/features/python.feature index 9fcb15efee..a01ecf3267 100644 --- a/extensions/python/tests/features/python.feature +++ b/extensions/python/tests/features/python.feature @@ -25,6 +25,8 @@ Feature: MiNiFi can use python processors in its flows And the "success" relationship of the GenerateFlowFile processor is connected to the AddPythonAttribute And the "success" relationship of the AddPythonAttribute processor is connected to the LogAttribute And AddPythonAttribute's success relationship is auto-terminated + And AddPythonAttribute's failure relationship is auto-terminated + And LogAttribute's success relationship is auto-terminated When all instances start up Then the Minifi logs contain the following message: "key:Python attribute value:attributevalue" in less than 60 seconds @@ -36,6 +38,7 @@ Feature: MiNiFi can use python processors in its flows And the "Dynamic Property" property of the LogDynamicProperties processor is set to "dynamic value" And the "success" relationship of the GenerateFlowFile processor is connected to the LogDynamicProperties And LogDynamicProperties's success relationship is auto-terminated + And LogDynamicProperties's failure relationship is auto-terminated When all instances start up Then the Minifi logs contain the following message: "Static Property value: static value" in less than 60 seconds @@ -53,7 +56,9 @@ Feature: MiNiFi can use python processors in its flows And PutFile is EVENT_DRIVEN And the "success" relationship of the GenerateFlowFile processor is connected to the MoveContentToJson And the "success" relationship of the MoveContentToJson processor is connected to the PutFile + And MoveContentToJson's failure relationship is auto-terminated And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When all instances start up Then a single file with the content '{"content": ""}' is placed in the '/tmp/output' directory in less than 60 seconds @@ -65,6 +70,7 @@ Feature: MiNiFi can use python processors in its flows And RemoveFlowFile is EVENT_DRIVEN And the "success" relationship of the GenerateFlowFile processor is connected to the RemoveFlowFile And RemoveFlowFile's success relationship is auto-terminated + And RemoveFlowFile's failure relationship is auto-terminated When all instances start up Then the Minifi logs contain the following message: "Removing flow file with UUID" in less than 30 seconds @@ -76,7 +82,9 @@ Feature: MiNiFi can use python processors in its flows And a PutFile processor with the "Directory" property set to "/tmp/output" And PutFile is EVENT_DRIVEN And the "success" relationship of the CountingProcessor processor is connected to the PutFile + And CountingProcessor's failure relationship is auto-terminated And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When all instances start up Then files with at least these contents "1,2,3,4,5" are placed in the "/tmp/output" directory in less than 5 seconds @@ -101,6 +109,9 @@ Feature: MiNiFi can use python processors in its flows And the "success" relationship of the ParseDocument processor is connected to the ChunkDocument And the "success" relationship of the ChunkDocument processor is connected to the PutFile And the "success" relationship of the PutFile processor is connected to the LogAttribute + And ParseDocument's failure relationship is auto-terminated + And ChunkDocument's failure relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And LogAttribute's success relationship is auto-terminated And the MiNiFi deployment timeout is set to 300 seconds @@ -124,6 +135,9 @@ Feature: MiNiFi can use python processors in its flows And LogAttribute is EVENT_DRIVEN And the "space" relationship of the CreateFlowFile processor is connected to the PutFile And the "success" relationship of the PutFile processor is connected to the LogAttribute + And CreateFlowFile's success relationship is auto-terminated + And CreateFlowFile's failure relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And LogAttribute's success relationship is auto-terminated When the MiNiFi instance starts up Then a single file with the content 'Hello World!' is placed in the '/tmp/output' directory in less than 10 seconds @@ -145,6 +159,10 @@ Feature: MiNiFi can use python processors in its flows And the "second" relationship of the RotatingForwarder processor is connected to the PutFile And the "third" relationship of the RotatingForwarder processor is connected to the PutFile And the "fourth" relationship of the RotatingForwarder processor is connected to the PutFile + And RotatingForwarder's success relationship is auto-terminated + And RotatingForwarder's failure relationship is auto-terminated + And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When all instances start up @@ -161,7 +179,9 @@ Feature: MiNiFi can use python processors in its flows And the "success" relationship of the GenerateFlowFile processor is connected to the SpecialPropertyTypeChecker And the "success" relationship of the SpecialPropertyTypeChecker processor is connected to the PutFile + And SpecialPropertyTypeChecker's failure relationship is auto-terminated And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When all instances start up @@ -177,7 +197,10 @@ Feature: MiNiFi can use python processors in its flows And the "success" relationship of the GenerateFlowFile processor is connected to the ProcessContextInterfaceChecker And the "myrelationship" relationship of the ProcessContextInterfaceChecker processor is connected to the PutFile + And ProcessContextInterfaceChecker's success relationship is auto-terminated + And ProcessContextInterfaceChecker's failure relationship is auto-terminated And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When all instances start up @@ -196,6 +219,9 @@ Feature: MiNiFi can use python processors in its flows And the "success" relationship of the GenerateFlowFile processor is connected to the UpdateAttribute And the "success" relationship of the UpdateAttribute processor is connected to the FailureWithAttributes And the "failure" relationship of the FailureWithAttributes processor is connected to the LogAttribute + And UpdateAttribute's failure relationship is auto-terminated + And FailureWithAttributes's success relationship is auto-terminated + And FailureWithAttributes's failure relationship is auto-terminated And LogAttribute's success relationship is auto-terminated When all instances start up @@ -213,7 +239,9 @@ Feature: MiNiFi can use python processors in its flows And the "success" relationship of the GenerateFlowFile processor is connected to the RelativeImporterProcessor And the "success" relationship of the RelativeImporterProcessor processor is connected to the PutFile + And RelativeImporterProcessor's failure relationship is auto-terminated And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When all instances start up @@ -223,8 +251,12 @@ Feature: MiNiFi can use python processors in its flows Given a org.apache.nifi.minifi.processors.nifi_python_processors.CreateNothing processor with the name "CreateNothing" And a PutFile processor with the "Directory" property set to "/tmp/output" And PutFile is EVENT_DRIVEN + And the "success" relationship of the CreateNothing processor is connected to the PutFile + And CreateNothing's failure relationship is auto-terminated And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated + When the MiNiFi instance starts up Then no files are placed in the "/tmp/output" directory in 10 seconds of running time And the Minifi logs do not contain the following message: "Caught Exception during SchedulingAgent::onTrigger of processor CreateNothing" after 1 seconds @@ -235,6 +267,8 @@ Feature: MiNiFi can use python processors in its flows And FailureWithContent is EVENT_DRIVEN And the "success" relationship of the GenerateFlowFile processor is connected to the FailureWithContent + And FailureWithContent's success relationship is auto-terminated + And FailureWithContent's failure relationship is auto-terminated When all instances start up @@ -246,6 +280,8 @@ Feature: MiNiFi can use python processors in its flows And TransferToOriginal is EVENT_DRIVEN And the "success" relationship of the GenerateFlowFile processor is connected to the TransferToOriginal + And TransferToOriginal's success relationship is auto-terminated + And TransferToOriginal's failure relationship is auto-terminated When all instances start up @@ -267,6 +303,7 @@ Feature: MiNiFi can use python processors in its flows And the "success" relationship of the GetFile processor is connected to the SetRecordField And the "success" relationship of the SetRecordField processor is connected to the LogAttribute + And SetRecordField's failure relationship is auto-terminated And LogAttribute's success relationship is auto-terminated When all instances start up @@ -282,6 +319,7 @@ Feature: MiNiFi can use python processors in its flows And a LogAttribute processor with the "FlowFiles To Log" property set to "0" And LogAttribute is EVENT_DRIVEN And the "success" relationship of the TestStateManager processor is connected to the LogAttribute + And TestStateManager's failure relationship is auto-terminated And LogAttribute's success relationship is auto-terminated When the MiNiFi instance starts up Then the Minifi logs contain the following message: "key:state_key value:1" in less than 10 seconds @@ -295,6 +333,7 @@ Feature: MiNiFi can use python processors in its flows And the "Dynamic Property" property of the NifiStyleLogDynamicProperties processor is set to "dynamic value" And the "success" relationship of the GenerateFlowFile processor is connected to the NifiStyleLogDynamicProperties And NifiStyleLogDynamicProperties's success relationship is auto-terminated + And NifiStyleLogDynamicProperties's failure relationship is auto-terminated When all instances start up Then the Minifi logs contain the following message: "Static Property value: static value" in less than 10 seconds @@ -307,11 +346,12 @@ Feature: MiNiFi can use python processors in its flows And the "Integer Property" property of the ExpressionLanguagePropertyWithValidator processor is set to "42" And the "success" relationship of the GenerateFlowFile processor is connected to the ExpressionLanguagePropertyWithValidator And ExpressionLanguagePropertyWithValidator's success relationship is auto-terminated + And ExpressionLanguagePropertyWithValidator's failure relationship is auto-terminated When all instances start up Then the Minifi logs contain the following message: "Integer Property value: 42" in less than 10 seconds - Scenario: MiNiFi C++ rolls back session if expression language cannot be evaluated as integer in NiFi python processors + Scenario: MiNiFi C++ transfers to failure if expression language cannot be evaluated as integer in NiFi python processors Given a GenerateFlowFile processor with the "File Size" property set to "0B" And a UpdateAttribute processor with the "my.integer" property set to "invalid" And UpdateAttribute is EVENT_DRIVEN @@ -320,10 +360,12 @@ Feature: MiNiFi can use python processors in its flows And the "Integer Property" property of the ExpressionLanguagePropertyWithValidator processor is set to "${my.integer}" And the "success" relationship of the GenerateFlowFile processor is connected to the UpdateAttribute And the "success" relationship of the UpdateAttribute processor is connected to the ExpressionLanguagePropertyWithValidator + And UpdateAttribute's failure relationship is auto-terminated And ExpressionLanguagePropertyWithValidator's success relationship is auto-terminated + And ExpressionLanguagePropertyWithValidator's failure relationship is auto-terminated When all instances start up - Then the Minifi logs contain the following message: "ProcessSession rollback for ExpressionLanguagePropertyWithValidator" in less than 10 seconds + Then the Minifi logs contain the following message: "from ExpressionLanguagePropertyWithValidator to relationship failure" in less than 10 seconds Scenario: MiNiFi C++ can use evaluate expression language expressions correctly using the NiFi python API Given a GenerateFlowFile processor with the "File Size" property set to "0B" @@ -341,7 +383,10 @@ Feature: MiNiFi can use python processors in its flows And the "success" relationship of the GenerateFlowFile processor is connected to the UpdateAttribute And the "success" relationship of the UpdateAttribute processor is connected to the EvaluateExpressionLanguageChecker And the "success" relationship of the EvaluateExpressionLanguageChecker processor is connected to the PutFile + And UpdateAttribute's failure relationship is auto-terminated + And EvaluateExpressionLanguageChecker's failure relationship is auto-terminated And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When all instances start up @@ -359,6 +404,7 @@ Feature: MiNiFi can use python processors in its flows And the max concurrent tasks attribute of the SleepForever processor is set to And the scheduling period of the SleepForever processor is set to "100 milliseconds" And SleepForever's success relationship is auto-terminated + And SleepForever's failure relationship is auto-terminated When the MiNiFi instance starts up @@ -374,6 +420,7 @@ Feature: MiNiFi can use python processors in its flows And the max concurrent tasks attribute of the SingleThreadedSleepForever processor is set to 5 And the scheduling period of the SingleThreadedSleepForever processor is set to "100 milliseconds" And SingleThreadedSleepForever's success relationship is auto-terminated + And SingleThreadedSleepForever's failure relationship is auto-terminated When the MiNiFi instance starts up @@ -384,6 +431,7 @@ Feature: MiNiFi can use python processors in its flows And the max concurrent tasks attribute of the MinifiSleepForever processor is set to And the scheduling period of the MinifiSleepForever processor is set to "100 milliseconds" And MinifiSleepForever's success relationship is auto-terminated + And MinifiSleepForever's failure relationship is auto-terminated When the MiNiFi instance starts up @@ -399,6 +447,7 @@ Feature: MiNiFi can use python processors in its flows And the max concurrent tasks attribute of the SingleThreadedMinifiSleepForever processor is set to 5 And the scheduling period of the SingleThreadedMinifiSleepForever processor is set to "100 milliseconds" And SingleThreadedMinifiSleepForever's success relationship is auto-terminated + And SingleThreadedMinifiSleepForever's failure relationship is auto-terminated When the MiNiFi instance starts up diff --git a/extensions/splunk/tests/features/splunk.feature b/extensions/splunk/tests/features/splunk.feature index 81a5fe4ab9..ae0ec576e8 100644 --- a/extensions/splunk/tests/features/splunk.feature +++ b/extensions/splunk/tests/features/splunk.feature @@ -43,6 +43,10 @@ Feature: Sending data to Splunk HEC using PutSplunkHTTP And the "undetermined" relationship of the QuerySplunkIndexingStatus processor is connected to the QuerySplunkIndexingStatus And the "acknowledged" relationship of the QuerySplunkIndexingStatus processor is connected to the PutFile And PutFile's success relationship is auto-terminated + And PutSplunkHTTP's failure relationship is auto-terminated + And QuerySplunkIndexingStatus's failure relationship is auto-terminated + And QuerySplunkIndexingStatus's unacknowledged relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When the MiNiFi instance starts up Then a single file with the content "foobar" is placed in the "/tmp/output" directory in less than 20 seconds @@ -76,6 +80,11 @@ Feature: Sending data to Splunk HEC using PutSplunkHTTP And the "undetermined" relationship of the QuerySplunkIndexingStatus processor is connected to the QuerySplunkIndexingStatus And the "acknowledged" relationship of the QuerySplunkIndexingStatus processor is connected to the PutFile And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated + And PutSplunkHTTP's failure relationship is auto-terminated + And QuerySplunkIndexingStatus's failure relationship is auto-terminated + And QuerySplunkIndexingStatus's unacknowledged relationship is auto-terminated + And SSL is enabled for the Splunk HEC and the SSL context service is set up for PutSplunkHTTP and QuerySplunkIndexingStatus When the MiNiFi instance starts up diff --git a/extensions/sql/tests/features/sql.feature b/extensions/sql/tests/features/sql.feature index e2b54c20d1..d7c34ab3da 100644 --- a/extensions/sql/tests/features/sql.feature +++ b/extensions/sql/tests/features/sql.feature @@ -26,6 +26,7 @@ Feature: Executing SQL operations from MiNiFi-C++ And the "success" relationship of the GenerateFlowFile processor is connected to the UpdateAttribute And the "success" relationship of the UpdateAttribute processor is connected to the PutSQL And an ODBCService is setup up for PutSQL with the name "ODBCService" + And UpdateAttribute's failure relationship is auto-terminated And PutSQL's success relationship is auto-terminated And PutSQL's failure relationship is auto-terminated And a PostgreSQL server is set up @@ -42,6 +43,10 @@ Feature: Executing SQL operations from MiNiFi-C++ And the "success" relationship of the GenerateFlowFile processor is connected to the UpdateAttribute And the "success" relationship of the UpdateAttribute processor is connected to the ExecuteSQL And the "success" relationship of the ExecuteSQL processor is connected to the PutFile + And ExecuteSQL's failure relationship is auto-terminated + And UpdateAttribute's failure relationship is auto-terminated + And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And an ODBCService is setup up for ExecuteSQL with the name "ODBCService" And a PostgreSQL server is set up When all instances start up @@ -58,6 +63,10 @@ Feature: Executing SQL operations from MiNiFi-C++ And the "success" relationship of the GenerateFlowFile processor is connected to the UpdateAttribute And the "success" relationship of the UpdateAttribute processor is connected to the ExecuteSQL And the "success" relationship of the ExecuteSQL processor is connected to the PutFile + And ExecuteSQL's failure relationship is auto-terminated + And UpdateAttribute's failure relationship is auto-terminated + And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And an ODBCService is setup up for ExecuteSQL with the name "ODBCService" And a PostgreSQL server is set up When all instances start up @@ -70,6 +79,8 @@ Feature: Executing SQL operations from MiNiFi-C++ And the "Output Format" property of the QueryDatabaseTable processor is set to "JSON" And a PutFile processor with the "Directory" property set to "/tmp/output" And the "success" relationship of the QueryDatabaseTable processor is connected to the PutFile + And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And an ODBCService is setup up for QueryDatabaseTable with the name "ODBCService" And a PostgreSQL server is set up When all instances start up @@ -83,6 +94,8 @@ Feature: Executing SQL operations from MiNiFi-C++ And the "Output Format" property of the QueryDatabaseTable processor is set to "JSON" And a PutFile processor with the "Directory" property set to "/tmp/output" And the "success" relationship of the QueryDatabaseTable processor is connected to the PutFile + And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And an ODBCService is setup up for QueryDatabaseTable with the name "ODBCService" And a PostgreSQL server is set up When all instances start up diff --git a/extensions/standard-processors/tests/features/attributes_to_json.feature b/extensions/standard-processors/tests/features/attributes_to_json.feature index 7f68c0b63d..e5143624cf 100644 --- a/extensions/standard-processors/tests/features/attributes_to_json.feature +++ b/extensions/standard-processors/tests/features/attributes_to_json.feature @@ -26,5 +26,6 @@ Feature: Writing attribute data using AttributesToJSON processor And the "success" relationship of the GetFile processor is connected to the AttributesToJSON And the "success" relationship of the AttributesToJSON processor is connected to the PutFile And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When the MiNiFi instance starts up Then a file with the JSON content "{"invalid":null,"filename":"test_file.log"}" is placed in the "/tmp/output" directory in less than 20 seconds diff --git a/extensions/standard-processors/tests/features/core_functionality.feature b/extensions/standard-processors/tests/features/core_functionality.feature index 768c13f5a5..d63718960b 100644 --- a/extensions/standard-processors/tests/features/core_functionality.feature +++ b/extensions/standard-processors/tests/features/core_functionality.feature @@ -30,6 +30,7 @@ Feature: Core flow functionalities And the "success" relationship of the Generate2 processor is connected to the Funnel1 And the Funnel with the name "Funnel1" is connected to the PutFile And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When all instances start up @@ -45,12 +46,14 @@ Feature: Core flow functionalities Scenario: The default configuration uses RocksDB for both the flow file and content repositories Given a GenerateFlowFile processor with the "Data Format" property set to "Text" + And GenerateFlowFile's success relationship is auto-terminated When the MiNiFi instance starts up Then the Minifi logs contain the following message: "Using plaintext FlowFileRepository" in less than 5 seconds And the Minifi logs contain the following message: "Using plaintext DatabaseContentRepository" in less than 1 second Scenario: Agent does not crash when using provenance repositories Given a GenerateFlowFile processor with the name "generateFlowFile" + And generateFlowFile's success relationship is auto-terminated And MiNiFi configuration "nifi.provenance.repository.class.name" is set to "VolatileProvenanceRepository" When the MiNiFi instance starts up Then the Minifi logs contain the following message: "MiNiFi started" in less than 40 seconds @@ -89,6 +92,8 @@ Feature: Core flow functionalities And the "success" relationship of the GetFile processor is connected to the UpdateAttribute And the "success" relationship of the UpdateAttribute processor is connected to the PutFile And the "success" relationship of the PutFile processor is connected to the LogAttribute + And UpdateAttribute's failure relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And LogAttribute's success relationship is auto-terminated When all instances start up @@ -109,7 +114,8 @@ Feature: Core flow functionalities And the "success" relationship of the GenerateFlowFile processor is connected to the ConvertRecord And a PutFile processor with the "Directory" property set to "/tmp/output" And the "success" relationship of the ConvertRecord processor is connected to the PutFile + And ConvertRecord's failure relationship is auto-terminated And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When the MiNiFi instance starts up Then at least one file with the content "[{"numbers":[1,2]}]" is placed in the "/tmp/output" directory in less than 10 seconds - diff --git a/extensions/standard-processors/tests/features/defragtextflowfiles.feature b/extensions/standard-processors/tests/features/defragtextflowfiles.feature index d61e2e97ff..83793712e7 100644 --- a/extensions/standard-processors/tests/features/defragtextflowfiles.feature +++ b/extensions/standard-processors/tests/features/defragtextflowfiles.feature @@ -31,7 +31,9 @@ Feature: DefragmentText can defragment fragmented data from TailFile And SuccessPut is EVENT_DRIVEN And the "success" relationship of the MultiTail processor is connected to the DefragmentText And the "success" relationship of the DefragmentText processor is connected to the SuccessPut + And DefragmentText's failure relationship is auto-terminated And SuccessPut's success relationship is auto-terminated + And SuccessPut's failure relationship is auto-terminated When all instances start up Then files with contents "" are placed in the "/tmp/output" directory in less than 60 seconds @@ -58,7 +60,9 @@ Feature: DefragmentText can defragment fragmented data from TailFile And the "success" relationship of the TailOne processor is connected to the DefragmentText And the "success" relationship of the TailTwo processor is connected to the DefragmentText And the "success" relationship of the DefragmentText processor is connected to the SuccessPut + And DefragmentText's failure relationship is auto-terminated And SuccessPut's success relationship is auto-terminated + And SuccessPut's failure relationship is auto-terminated When all instances start up Then files with contents "" are placed in the "/tmp/output" directory in less than 60 seconds @@ -80,7 +84,9 @@ Feature: DefragmentText can defragment fragmented data from TailFile And SuccessPut is EVENT_DRIVEN And the "success" relationship of the TailFile processor is connected to the DefragmentText And the "success" relationship of the DefragmentText processor is connected to the SuccessPut + And DefragmentText's failure relationship is auto-terminated And SuccessPut's success relationship is auto-terminated + And SuccessPut's failure relationship is auto-terminated When all instances start up Then files with contents "" are placed in the "/tmp/output" directory in less than 30 seconds diff --git a/extensions/standard-processors/tests/features/evaluate_json_path.feature b/extensions/standard-processors/tests/features/evaluate_json_path.feature index c87f91c424..84dc04e007 100644 --- a/extensions/standard-processors/tests/features/evaluate_json_path.feature +++ b/extensions/standard-processors/tests/features/evaluate_json_path.feature @@ -25,8 +25,11 @@ Feature: Writing JSON path query result to attribute or flow file using Evaluate And PutFile is EVENT_DRIVEN And the "success" relationship of the GetFile processor is connected to the EvaluateJsonPath And the "matched" relationship of the EvaluateJsonPath processor is connected to the PutFile + And EvaluateJsonPath's failure relationship is auto-terminated + And EvaluateJsonPath's unmatched relationship is auto-terminated And PutFile's success relationship is auto-terminated And PutFile's failure relationship is auto-terminated + When the MiNiFi instance starts up Then a file with the JSON content "["The Great Gatsby","1984"]" is placed in the "/tmp/output" directory in less than 10 seconds @@ -46,8 +49,11 @@ Feature: Writing JSON path query result to attribute or flow file using Evaluate And the "success" relationship of the GetFile processor is connected to the EvaluateJsonPath And the "matched" relationship of the EvaluateJsonPath processor is connected to the PutFile And the "success" relationship of the PutFile processor is connected to the LogAttribute + And EvaluateJsonPath's failure relationship is auto-terminated + And EvaluateJsonPath's unmatched relationship is auto-terminated And PutFile's failure relationship is auto-terminated And LogAttribute's success relationship is auto-terminated + When the MiNiFi instance starts up Then a file with the JSON content "{"title": "1984", "author": null}" is placed in the "/tmp/output" directory in less than 10 seconds And the Minifi logs contain the following message: "key:title value:1984" in less than 10 seconds diff --git a/extensions/standard-processors/tests/features/fetch_modbus_tcp.feature b/extensions/standard-processors/tests/features/fetch_modbus_tcp.feature index 779fd7bd86..3f0a309e2a 100644 --- a/extensions/standard-processors/tests/features/fetch_modbus_tcp.feature +++ b/extensions/standard-processors/tests/features/fetch_modbus_tcp.feature @@ -39,7 +39,9 @@ Feature: Minifi C++ can act as a modbus tcp master And the "foo" property of the FetchModbusTcp processor is set to "holding-register:52" And the "bar" property of the FetchModbusTcp processor is set to "405678:REAL" And the "baz" property of the FetchModbusTcp processor is set to "4x4444:CHAR[6]" + And FetchModbusTcp's failure relationship is auto-terminated And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When the MiNiFi instance starts up Then at least one file with the JSON content "{"foo":123,"bar":1.75,"baz":["M", "i", "N", "i", "F", "i"]}" is placed in the "/tmp/output" directory in less than 10 seconds diff --git a/extensions/standard-processors/tests/features/file_system_operations.feature b/extensions/standard-processors/tests/features/file_system_operations.feature index 5bec6a6f44..b498d91c7d 100644 --- a/extensions/standard-processors/tests/features/file_system_operations.feature +++ b/extensions/standard-processors/tests/features/file_system_operations.feature @@ -26,6 +26,7 @@ Feature: File system operations are handled by the GetFile, PutFile, ListFile an And PutFile is EVENT_DRIVEN And the "success" relationship of the GetFile processor is connected to the PutFile And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When the MiNiFi instance starts up Then a single file with the content "test" is placed in the "/tmp/output" directory in less than 10 seconds @@ -70,6 +71,10 @@ Feature: File system operations are handled by the GetFile, PutFile, ListFile an And PutFile is EVENT_DRIVEN And the "success" relationship of the ListFile processor is connected to the FetchFile And the "success" relationship of the FetchFile processor is connected to the PutFile + And FetchFile's failure relationship is auto-terminated + And FetchFile's not.found relationship is auto-terminated + And FetchFile's permission.denied relationship is auto-terminated And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When the MiNiFi instance starts up Then files with contents "Test message" and "Another test message" are placed in the "/tmp/output" directory in less than 10 seconds diff --git a/extensions/standard-processors/tests/features/hashcontent.feature b/extensions/standard-processors/tests/features/hashcontent.feature index 096ba183f8..3758ac6c93 100644 --- a/extensions/standard-processors/tests/features/hashcontent.feature +++ b/extensions/standard-processors/tests/features/hashcontent.feature @@ -29,6 +29,7 @@ Feature: Hash value is added to Flowfiles by HashContent processor And LogAttribute is EVENT_DRIVEN And the "success" relationship of the GetFile processor is connected to the HashContent And the "success" relationship of the HashContent processor is connected to the LogAttribute + And HashContent's failure relationship is auto-terminated And LogAttribute's success relationship is auto-terminated When the MiNiFi instance starts up Then the Minifi logs contain the following message: "key:hash value:" in less than 10 seconds @@ -50,7 +51,9 @@ Feature: Hash value is added to Flowfiles by HashContent processor And PutFile is EVENT_DRIVEN And the "success" relationship of the GetFile processor is connected to the HashContent And the "failure" relationship of the HashContent processor is connected to the PutFile + And HashContent's success relationship is auto-terminated And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When the MiNiFi instance starts up Then at least one empty file is placed in the "/tmp/output" directory in less than 10 seconds @@ -65,6 +68,7 @@ Feature: Hash value is added to Flowfiles by HashContent processor And LogAttribute is EVENT_DRIVEN And the "success" relationship of the GetFile processor is connected to the HashContent And the "success" relationship of the HashContent processor is connected to the LogAttribute + And HashContent's failure relationship is auto-terminated And LogAttribute's success relationship is auto-terminated When the MiNiFi instance starts up Then the Minifi logs contain the following message: "key:hash value:1F3870BE274F6C49B3E31A0C6728957F" in less than 10 seconds diff --git a/extensions/standard-processors/tests/features/minifi_c2_server.feature b/extensions/standard-processors/tests/features/minifi_c2_server.feature index c35ddeed95..9198d8744c 100644 --- a/extensions/standard-processors/tests/features/minifi_c2_server.feature +++ b/extensions/standard-processors/tests/features/minifi_c2_server.feature @@ -18,6 +18,7 @@ Feature: MiNiFi can communicate with Apache NiFi MiNiFi C2 server Scenario: MiNiFi flow config is updated from MiNiFi C2 server Given a GetFile processor with the name "GetFile1" and the "Input Directory" property set to "/tmp/non-existent" + And GetFile1's success relationship is auto-terminated And C2 is enabled in MiNiFi And a directory at "/tmp/input" has a file with the content "test" And a MiNiFi C2 server is set up @@ -38,6 +39,7 @@ Feature: MiNiFi can communicate with Apache NiFi MiNiFi C2 server Scenario: MiNiFi flow config is updated from MiNiFi C2 server through SSL with SSL properties Given a directory at "/tmp/input" has a file with the content "test" And a GenerateFlowFile processor + And GenerateFlowFile's success relationship is auto-terminated And ssl properties are set up for MiNiFi C2 server And a MiNiFi C2 server is set up with SSL When all instances start up @@ -46,6 +48,7 @@ Feature: MiNiFi can communicate with Apache NiFi MiNiFi C2 server Scenario: MiNiFi flow config is updated from MiNiFi C2 server with overriden parameter context Given a GetFile processor with the name "GetFile1" and the "Input Directory" property set to "${INPUT_DIR}" + And GetFile1's success relationship is auto-terminated And parameter context name is set to 'my-context' And a non-sensitive parameter in the flow config called 'INPUT_DIR' with the value '/tmp/non-existent' in the parameter context 'my-context' And C2 is enabled in MiNiFi diff --git a/extensions/standard-processors/tests/features/minifi_controller.feature b/extensions/standard-processors/tests/features/minifi_controller.feature index a5eba8bcfd..bc08442e70 100644 --- a/extensions/standard-processors/tests/features/minifi_controller.feature +++ b/extensions/standard-processors/tests/features/minifi_controller.feature @@ -21,6 +21,7 @@ Feature: MiNiFi Controller functionalities Given a GenerateFlowFile processor And a directory at "/tmp/input" has a file with the content "test" And controller socket properties are set up + And GenerateFlowFile's success relationship is auto-terminated When all instances start up And MiNiFi config is updated through MiNiFi controller Then a single file with the content "test" is placed in the "/tmp/output" directory in less than 60 seconds @@ -30,6 +31,7 @@ Feature: MiNiFi Controller functionalities Given a GenerateFlowFile processor And a directory at "/tmp/input" has a file with the content "test" And controller socket properties are set up + And GenerateFlowFile's success relationship is auto-terminated When all instances start up And the GenerateFlowFile component is stopped through MiNiFi controller Then the GenerateFlowFile component is not running @@ -39,6 +41,7 @@ Feature: MiNiFi Controller functionalities Given a GenerateFlowFile processor And a directory at "/tmp/input" has a file with the content "test" And controller socket properties are set up + And GenerateFlowFile's success relationship is auto-terminated When all instances start up And the FlowController component is stopped through MiNiFi controller Then the GenerateFlowFile component is not running @@ -48,6 +51,7 @@ Feature: MiNiFi Controller functionalities Given a GenerateFlowFile processor And a directory at "/tmp/input" has a file with the content "test" And controller socket properties are set up + And GenerateFlowFile's success relationship is auto-terminated When all instances start up And the FlowController component is stopped through MiNiFi controller And the FlowController component is started through MiNiFi controller @@ -58,6 +62,7 @@ Feature: MiNiFi Controller functionalities Given a GenerateFlowFile processor And a LogAttribute processor with the "FlowFiles To Log" property set to "0" And the "success" relationship of the GenerateFlowFile processor is connected to the LogAttribute + And LogAttribute's success relationship is auto-terminated And controller socket properties are set up When all instances start up And MiNiFi config is updated through MiNiFi controller @@ -69,6 +74,7 @@ Feature: MiNiFi Controller functionalities Given a GenerateFlowFile processor And a directory at "/tmp/input" has a file with the content "test" And controller socket properties are set up + And GenerateFlowFile's success relationship is auto-terminated When all instances start up Then manifest can be retrieved through MiNiFi controller @@ -76,5 +82,6 @@ Feature: MiNiFi Controller functionalities Given a GenerateFlowFile processor And a directory at "/tmp/input" has a file with the content "test" And controller socket properties are set up + And GenerateFlowFile's success relationship is auto-terminated When all instances start up Then debug bundle can be retrieved through MiNiFi controller diff --git a/extensions/standard-processors/tests/features/network_listener.feature b/extensions/standard-processors/tests/features/network_listener.feature index 4eb6f2bb10..6f6d1cd82e 100644 --- a/extensions/standard-processors/tests/features/network_listener.feature +++ b/extensions/standard-processors/tests/features/network_listener.feature @@ -24,6 +24,7 @@ Feature: Minifi C++ can act as a network listener And a TCP client is set up to send a test TCP message to minifi And the "success" relationship of the ListenTCP processor is connected to the PutFile And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When all instances start up Then at least one file with the content "test_tcp_message" is placed in the "/tmp/output" directory in less than 20 seconds diff --git a/extensions/standard-processors/tests/features/replace_text.feature b/extensions/standard-processors/tests/features/replace_text.feature index b24a0e8738..efda4cfbdc 100644 --- a/extensions/standard-processors/tests/features/replace_text.feature +++ b/extensions/standard-processors/tests/features/replace_text.feature @@ -28,6 +28,9 @@ Feature: Changing flowfile contents using the ReplaceText processor And a PutFile processor with the "Directory" property set to "/tmp/output" And the "success" relationship of the GenerateFlowFile processor is connected to the ReplaceText And the "success" relationship of the ReplaceText processor is connected to the PutFile + And ReplaceText's failure relationship is auto-terminated + And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When the MiNiFi instance starts up Then a single file with the content "" is placed in the "/tmp/output" directory in less than 10 seconds @@ -50,6 +53,9 @@ Feature: Changing flowfile contents using the ReplaceText processor And a PutFile processor with the "Directory" property set to "/tmp/output" And the "success" relationship of the GetFile processor is connected to the ReplaceText And the "success" relationship of the ReplaceText processor is connected to the PutFile + And ReplaceText's failure relationship is auto-terminated + And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When the MiNiFi instance starts up Then a single file with the content "" is placed in the "/tmp/output" directory in less than 10 seconds diff --git a/extensions/standard-processors/tests/features/routetext.feature b/extensions/standard-processors/tests/features/routetext.feature index 5a27f1b197..087eb8da4b 100644 --- a/extensions/standard-processors/tests/features/routetext.feature +++ b/extensions/standard-processors/tests/features/routetext.feature @@ -26,12 +26,17 @@ Feature: Processing log files line-by-line using RouteText And a UpdateAttribute processor with the name "UpdateInfo" and the "filename" property set to "info.txt" And a UpdateAttribute processor with the name "UpdateWarning" and the "filename" property set to "warning.txt" And a PutFile processor with the "Directory" property set to "/tmp/output" - And RouteText's original relationship is auto-terminated - And PutFile's success relationship is auto-terminated And the "success" relationship of the GetFile processor is connected to the RouteText And the "Info" relationship of the RouteText processor is connected to the UpdateInfo And the "Warning" relationship of the RouteText processor is connected to the UpdateWarning And the "success" relationship of the UpdateInfo processor is connected to the PutFile And the "success" relationship of the UpdateWarning processor is connected to the PutFile + And RouteText's original relationship is auto-terminated + And RouteText's matched relationship is auto-terminated + And RouteText's unmatched relationship is auto-terminated + And UpdateInfo's failure relationship is auto-terminated + And UpdateWarning's failure relationship is auto-terminated + And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When the MiNiFi instance starts up Then files with contents "[INFO] one\n[INFO] three\n" and "[WARNING] two\n[WARNING] four\n" are placed in the "/tmp/output" directory in less than 15 seconds diff --git a/extensions/standard-processors/tests/features/s2s.feature b/extensions/standard-processors/tests/features/s2s.feature index 89ebd58c63..a20759cd16 100644 --- a/extensions/standard-processors/tests/features/s2s.feature +++ b/extensions/standard-processors/tests/features/s2s.feature @@ -173,6 +173,7 @@ Feature: Sending data from MiNiFi-C++ to NiFi using S2S protocol And PutFile is EVENT_DRIVEN And the output port "from_nifi" is connected to the PutFile processor And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And a NiFi container is set up And a directory at "/tmp/input" has a file with the content "test" in the "nifi" flow @@ -193,6 +194,7 @@ Feature: Sending data from MiNiFi-C++ to NiFi using S2S protocol And PutFile is EVENT_DRIVEN And the output port "from_nifi" is connected to the PutFile processor And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And a NiFi container is set up And a directory at "/tmp/input" has a file with the content "test" in the "nifi" flow @@ -215,6 +217,7 @@ Feature: Sending data from MiNiFi-C++ to NiFi using S2S protocol And the output port "from_nifi" is connected to the PutFile processor And SSL properties are set in MiNiFi And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And a NiFi container is set up with SSL enabled And a directory at "/tmp/input" has a file with the content "test" in the "nifi" flow @@ -236,6 +239,7 @@ Feature: Sending data from MiNiFi-C++ to NiFi using S2S protocol And the output port "from_nifi" is connected to the PutFile processor And SSL properties are set in MiNiFi And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And a NiFi container is set up with SSL enabled And a directory at "/tmp/input" has a file with the content "test" in the "nifi" flow @@ -277,6 +281,7 @@ Feature: Sending data from MiNiFi-C++ to NiFi using S2S protocol And PutFile is EVENT_DRIVEN And the output port "from_nifi" is connected to the PutFile processor And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And a NiFi container is set up And a directory at "/tmp/input" has a file with the content "test" in the "nifi" flow @@ -297,6 +302,7 @@ Feature: Sending data from MiNiFi-C++ to NiFi using S2S protocol And PutFile is EVENT_DRIVEN And the output port "from_nifi" is connected to the PutFile processor And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And a NiFi container is set up And a directory at "/tmp/input" has a file with the content "test" in the "nifi" flow diff --git a/extensions/standard-processors/tests/features/split_json.feature b/extensions/standard-processors/tests/features/split_json.feature index 887d3832af..7e72c4a4fe 100644 --- a/extensions/standard-processors/tests/features/split_json.feature +++ b/extensions/standard-processors/tests/features/split_json.feature @@ -30,7 +30,11 @@ Feature: Splitting JSON content using SplitJson processor And the "split" relationship of the SplitJson processor is connected to the PutFile And the "original" relationship of the SplitJson processor is connected to the PutFile And the "success" relationship of the PutFile processor is connected to the LogAttribute + And SplitJson's failure relationship is auto-terminated + And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated And LogAttribute's success relationship is auto-terminated + When the MiNiFi instance starts up Then at least one file with the content "["Alice","Bob"]" is placed in the "/tmp/output" directory in less than 10 seconds And at least one file with the content "Dave" is placed in the "/tmp/output" directory in less than 1 seconds diff --git a/extensions/standard-processors/tests/features/splittext.feature b/extensions/standard-processors/tests/features/splittext.feature index aefe407af8..7d02546c7b 100644 --- a/extensions/standard-processors/tests/features/splittext.feature +++ b/extensions/standard-processors/tests/features/splittext.feature @@ -27,6 +27,9 @@ Feature: Split input text line-by-line using SplitText And PutFile is EVENT_DRIVEN And the "success" relationship of the GetFile processor is connected to the SplitText And the "splits" relationship of the SplitText processor is connected to the PutFile + And SplitText's failure relationship is auto-terminated + And PutFile's failure relationship is auto-terminated + When the MiNiFi instance starts up Then files with contents "[HEADER]header line 1\n[HEADER]header line 2\nDATA LINE 1,[HEADER]header line 1\n[HEADER]header line 2\nDATA LINE 2,[HEADER]header line 1\n[HEADER]header line 2" are placed in the "/tmp/output" directory in less than 15 seconds @@ -41,5 +44,8 @@ Feature: Split input text line-by-line using SplitText And the "splits" relationship of the SplitText processor is connected to the PutFile And SplitText's original relationship is auto-terminated And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated + And SplitText's failure relationship is auto-terminated + When the MiNiFi instance starts up Then files with contents "[HEADER]header line 1\n[HEADER]header line 2\nA BIT LONGER DATA LINE 1,[HEADER]header line 1\n[HEADER]header line 2\nDATA 2\nDATA 3" are placed in the "/tmp/output" directory in less than 15 seconds diff --git a/extensions/standard-processors/tests/features/syslog_listener.feature b/extensions/standard-processors/tests/features/syslog_listener.feature index 2ea0ec9909..c1b0b242ef 100644 --- a/extensions/standard-processors/tests/features/syslog_listener.feature +++ b/extensions/standard-processors/tests/features/syslog_listener.feature @@ -27,7 +27,10 @@ Feature: Minifi C++ can act as a syslog listener And the "Replacement Value" property of the ReplaceText processor is set to "${syslog.timestamp} ${syslog.priority} ${reverseDnsLookup(${syslog.sender}, 100)} ${syslog.msg}" And the "success" relationship of the ListenSyslog processor is connected to the ReplaceText And the "success" relationship of the ReplaceText processor is connected to the PutFile + And ListenSyslog's invalid relationship is auto-terminated + And ReplaceText's failure relationship is auto-terminated And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When both instances start up Then at least one file in "/tmp/output" content match the following regex: "^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.*13.*sample_log$" in less than 10 seconds @@ -39,7 +42,9 @@ Feature: Minifi C++ can act as a syslog listener And the "Parse Messages" property of the ListenSyslog processor is set to "true" And a Syslog client with TCP protocol is setup to send logs to minifi And the "success" relationship of the ListenSyslog processor is connected to the PutFile + And ListenSyslog's invalid relationship is auto-terminated And PutFile's success relationship is auto-terminated + And PutFile's failure relationship is auto-terminated When both instances start up Then at least 1 file is placed in the "/tmp/output" directory in less than 10 seconds diff --git a/extensions/standard-processors/tests/unit/AdaptiveConfigurationTests.cpp b/extensions/standard-processors/tests/unit/AdaptiveConfigurationTests.cpp index 4854240f53..a0b3a193d8 100644 --- a/extensions/standard-processors/tests/unit/AdaptiveConfigurationTests.cpp +++ b/extensions/standard-processors/tests/unit/AdaptiveConfigurationTests.cpp @@ -93,3 +93,39 @@ TEST_CASE("Adaptive configuration logs json parse errors") { REQUIRE(minifi::test::utils::verifyLogLinePresenceInPollTime(0s, "[error] Configuration file is not valid json: Invalid encoding in string. (38)")); REQUIRE(minifi::test::utils::verifyLogLinePresenceInPollTime(0s, "[error] Configuration file is not valid yaml: yaml-cpp: error at line 3, column 27: end of map flow not found")); } + +TEST_CASE("Adaptive configuration rejects JSON flow with unterminated relationships") { + ConfigurationTestController controller; + + const char* json_config = R"( + { + "Flow Controller": {"name": "root"}, + "Processors": [ + { + "id": "00000000-0000-0000-0000-000000000001", + "class": "GenerateFlowFile", + "name": "GenerateFlowFile" + }, + { + "id": "00000000-0000-0000-0000-000000000002", + "class": "LogAttribute", + "name": "LogAttribute" + } + ], + "Connections": [ + { + "id": "00000000-0000-0000-0000-000000000005", + "name": "GenerateFlowFileToLogAttribute", + "source id": "00000000-0000-0000-0000-000000000001", + "source relationship names": ["success"], + "destination id": "00000000-0000-0000-0000-000000000002" + } + ] + } + )"; + + core::flow::AdaptiveConfiguration config{controller.getContext()}; + + REQUIRE_THROWS_WITH(config.getRootFromPayload(json_config), + "General Operation: The following component-relationship pairs are not terminated by any connection: [(LogAttribute, success)]"); +} diff --git a/extensions/standard-processors/tests/unit/FlowJsonTests.cpp b/extensions/standard-processors/tests/unit/FlowJsonTests.cpp index 4fe379fdfa..861714a2c1 100644 --- a/extensions/standard-processors/tests/unit/FlowJsonTests.cpp +++ b/extensions/standard-processors/tests/unit/FlowJsonTests.cpp @@ -81,7 +81,7 @@ TEST_CASE("NiFi flow json format is correctly parsed") { "yieldDuration": "4 sec", "bulletinLevel": "ERROR", "runDurationMillis": 12, - "autoTerminatedRelationships": ["one", "two"], + "autoTerminatedRelationships": ["one", "two", "success"], "properties": { "File Size": "#{file_size}", "Batch Size": "#{batch_size}", @@ -1822,7 +1822,7 @@ TEST_CASE("Output port can also be used in RPG") { "name": "PutFile", "type": "org.apache.nifi.processors.standard.PutFile", "schedulingStrategy": "Event_DRIVEN", - "autoTerminatedRelationships": ["success"], + "autoTerminatedRelationships": ["success", "failure"], "properties": {} }], "connections": [{ @@ -1872,4 +1872,341 @@ TEST_CASE("Output port can also be used in RPG") { CHECK(port.get().getTimeout().value() == std::chrono::seconds(19)); } +namespace { +std::string unterminatedRelationshipsError(std::string_view processor_relationship_pairs) { + return fmt::format("General Operation: The following component-relationship pairs are not terminated by any connection: {}", + processor_relationship_pairs); +} +} // namespace + +TEST_CASE("NiFi flow json with all relationships terminated is loaded", "[RelationshipTermination]") { + ConfigurationTestController test_controller; + core::flow::AdaptiveConfiguration config(test_controller.getContext()); + + static const std::string CONFIG_JSON = + R"( +{ + "rootGroup": { + "name": "MiNiFi Flow", + "processors": [{ + "identifier": "00000000-0000-0000-0000-000000000001", + "name": "GenerateFlowFile", + "type": "org.apache.nifi.processors.standard.GenerateFlowFile", + "schedulingStrategy": "TIMER_DRIVEN", + "schedulingPeriod": "10 sec" + }, { + "identifier": "00000000-0000-0000-0000-000000000002", + "name": "LogAttribute", + "type": "org.apache.nifi.processors.standard.LogAttribute", + "schedulingStrategy": "EVENT_DRIVEN", + "autoTerminatedRelationships": ["success"] + }], + "funnels": [{ + "identifier": "00000000-0000-0000-0000-000000000003", + "name": "MyFunnel" + }], + "connections": [{ + "identifier": "00000000-0000-0000-0000-000000000011", + "name": "GenerateFlowFileToFunnel", + "source": {"id": "00000000-0000-0000-0000-000000000001"}, + "destination": {"id": "00000000-0000-0000-0000-000000000003"}, + "selectedRelationships": ["success"] + }, { + "identifier": "00000000-0000-0000-0000-000000000012", + "name": "FunnelToChildInputPort", + "source": {"id": "00000000-0000-0000-0000-000000000003"}, + "destination": {"id": "00000000-0000-0000-0000-000000000005"}, + "selectedRelationships": [""] + }, { + "identifier": "00000000-0000-0000-0000-000000000013", + "name": "ChildOutputPortToLogAttribute", + "source": {"id": "00000000-0000-0000-0000-000000000006"}, + "destination": {"id": "00000000-0000-0000-0000-000000000002"}, + "selectedRelationships": [""] + }, { + "identifier": "00000000-0000-0000-0000-000000000014", + "name": "RPGOutputPortToLogAttribute", + "source": {"id": "00000000-0000-0000-0000-000000000008"}, + "destination": {"id": "00000000-0000-0000-0000-000000000002"}, + "selectedRelationships": [] + }, { + "identifier": "00000000-0000-0000-0000-000000000015", + "name": "GenerateFlowFileToRPGInputPort", + "source": {"id": "00000000-0000-0000-0000-000000000001"}, + "destination": {"id": "00000000-0000-0000-0000-000000000009"}, + "selectedRelationships": ["success"] + }], + "processGroups": [{ + "identifier": "00000000-0000-0000-0000-000000000004", + "name": "Child Process Group", + "processors": [{ + "identifier": "00000000-0000-0000-0000-000000000007", + "name": "UpdateAttribute", + "type": "org.apache.nifi.processors.attributes.UpdateAttribute", + "schedulingStrategy": "EVENT_DRIVEN", + "autoTerminatedRelationships": ["failure"] + }], + "inputPorts": [{ + "identifier": "00000000-0000-0000-0000-000000000005", + "name": "in" + }], + "outputPorts": [{ + "identifier": "00000000-0000-0000-0000-000000000006", + "name": "out" + }], + "connections": [{ + "identifier": "00000000-0000-0000-0000-000000000016", + "name": "InputPortToUpdateAttribute", + "source": {"id": "00000000-0000-0000-0000-000000000005"}, + "destination": {"id": "00000000-0000-0000-0000-000000000007"}, + "selectedRelationships": [""] + }, { + "identifier": "00000000-0000-0000-0000-000000000017", + "name": "UpdateAttributeToOutputPort", + "source": {"id": "00000000-0000-0000-0000-000000000007"}, + "destination": {"id": "00000000-0000-0000-0000-000000000006"}, + "selectedRelationships": ["success"] + }] + }], + "remoteProcessGroups": [{ + "name": "NiFi Flow", + "targetUri": "https://localhost:8090/nifi", + "inputPorts": [{ + "identifier": "00000000-0000-0000-0000-000000000009", + "name": "AmazingInputPort", + "targetId": "00000000-0000-0000-0000-000000000019" + }], + "outputPorts": [{ + "identifier": "00000000-0000-0000-0000-000000000008", + "name": "AmazingOutputPort", + "targetId": "00000000-0000-0000-0000-000000000018" + }] + }] + } +})"; + + std::unique_ptr flow; + REQUIRE_NOTHROW(flow = config.getRootFromPayload(CONFIG_JSON)); + REQUIRE(flow); + + // connections from funnels and ports with an empty ("" or []) relationship list use the default relationship of the source + std::map connections; + flow->getConnections(connections); + CHECK(connections.at("FunnelToChildInputPort")->getRelationships() == (std::set{{"success", ""}})); + CHECK(connections.at("ChildOutputPortToLogAttribute")->getRelationships() == (std::set{{"success", ""}})); + CHECK(connections.at("InputPortToUpdateAttribute")->getRelationships() == (std::set{{"success", ""}})); + CHECK(connections.at("RPGOutputPortToLogAttribute")->getRelationships() == (std::set{{"undefined", ""}})); +} + +TEST_CASE("NiFi flow json with a processor relationship which is neither connected nor auto-terminated is rejected", "[RelationshipTermination]") { + ConfigurationTestController test_controller; + core::flow::AdaptiveConfiguration config(test_controller.getContext()); + + static const std::string CONFIG_JSON = + R"( +{ + "rootGroup": { + "name": "MiNiFi Flow", + "processors": [{ + "identifier": "00000000-0000-0000-0000-000000000001", + "name": "PutFile", + "type": "org.apache.nifi.processors.standard.PutFile", + "schedulingStrategy": "EVENT_DRIVEN", + "autoTerminatedRelationships": ["failure"] + }, { + "identifier": "00000000-0000-0000-0000-000000000002", + "name": "GenerateFlowFile", + "type": "org.apache.nifi.processors.standard.GenerateFlowFile", + "schedulingStrategy": "TIMER_DRIVEN", + "schedulingPeriod": "10 sec" + }], + "connections": [{ + "identifier": "00000000-0000-0000-0000-000000000003", + "name": "GenerateFlowFileToPutFile", + "source": {"id": "00000000-0000-0000-0000-000000000002"}, + "destination": {"id": "00000000-0000-0000-0000-000000000001"}, + "selectedRelationships": ["success"] + }] + } +})"; + + REQUIRE_THROWS_WITH(config.getRootFromPayload(CONFIG_JSON), unterminatedRelationshipsError("[(PutFile, success)]")); +} + +TEST_CASE("NiFi flow json connection with an empty relationship name does not terminate a processor relationship", "[RelationshipTermination]") { + ConfigurationTestController test_controller; + core::flow::AdaptiveConfiguration config(test_controller.getContext()); + + static const std::string CONFIG_JSON = + R"( +{ + "rootGroup": { + "name": "MiNiFi Flow", + "processors": [{ + "identifier": "00000000-0000-0000-0000-000000000001", + "name": "GenerateFlowFile", + "type": "org.apache.nifi.processors.standard.GenerateFlowFile", + "schedulingStrategy": "TIMER_DRIVEN", + "schedulingPeriod": "10 sec" + }, { + "identifier": "00000000-0000-0000-0000-000000000002", + "name": "LogAttribute", + "type": "org.apache.nifi.processors.standard.LogAttribute", + "schedulingStrategy": "EVENT_DRIVEN", + "autoTerminatedRelationships": ["success"] + }], + "connections": [{ + "identifier": "00000000-0000-0000-0000-000000000003", + "name": "GenerateFlowFileToLogAttribute", + "source": {"id": "00000000-0000-0000-0000-000000000001"}, + "destination": {"id": "00000000-0000-0000-0000-000000000002"}, + "selectedRelationships": [""] + }] + } +})"; + + REQUIRE_THROWS_WITH(config.getRootFromPayload(CONFIG_JSON), unterminatedRelationshipsError("[(GenerateFlowFile, success)]")); +} + +TEST_CASE("NiFi flow json with a funnel without outgoing connection is accepted, the funnel is used as a terminator", "[RelationshipTermination]") { + ConfigurationTestController test_controller; + core::flow::AdaptiveConfiguration config(test_controller.getContext()); + + static const std::string CONFIG_JSON = + R"( +{ + "rootGroup": { + "name": "MiNiFi Flow", + "processors": [{ + "identifier": "00000000-0000-0000-0000-000000000001", + "name": "GenerateFlowFile", + "type": "org.apache.nifi.processors.standard.GenerateFlowFile", + "schedulingStrategy": "TIMER_DRIVEN", + "schedulingPeriod": "10 sec" + }], + "funnels": [{ + "identifier": "00000000-0000-0000-0000-000000000002", + "name": "MyFunnel" + }], + "connections": [{ + "identifier": "00000000-0000-0000-0000-000000000003", + "name": "GenerateFlowFileToFunnel", + "source": {"id": "00000000-0000-0000-0000-000000000001"}, + "destination": {"id": "00000000-0000-0000-0000-000000000002"}, + "selectedRelationships": ["success"] + }] + } +})"; + + std::unique_ptr flow; + REQUIRE_NOTHROW(flow = config.getRootFromPayload(CONFIG_JSON)); + REQUIRE(flow); +} + +TEST_CASE("NiFi flow json with a remote process group output port without outgoing connection is rejected", "[RelationshipTermination]") { + ConfigurationTestController test_controller; + core::flow::AdaptiveConfiguration config(test_controller.getContext()); + + static const std::string CONFIG_JSON = + R"( +{ + "rootGroup": { + "name": "MiNiFi Flow", + "processors": [], + "connections": [], + "remoteProcessGroups": [{ + "name": "NiFi Flow", + "targetUri": "https://localhost:8090/nifi", + "outputPorts": [{ + "identifier": "00000000-0000-0000-0000-000000000001", + "name": "AmazingOutputPort", + "targetId": "00000000-0000-0000-0000-000000000002" + }] + }] + } +})"; + + REQUIRE_THROWS_WITH(config.getRootFromPayload(CONFIG_JSON), unterminatedRelationshipsError("[(AmazingOutputPort, undefined)]")); +} + +TEST_CASE("NiFi flow json with a process group output port without outgoing connection is rejected", "[RelationshipTermination]") { + ConfigurationTestController test_controller; + core::flow::AdaptiveConfiguration config(test_controller.getContext()); + + static const std::string CONFIG_JSON = + R"( +{ + "rootGroup": { + "name": "MiNiFi Flow", + "processors": [], + "connections": [], + "processGroups": [{ + "identifier": "00000000-0000-0000-0000-000000000001", + "name": "Child Process Group", + "processors": [{ + "identifier": "00000000-0000-0000-0000-000000000002", + "name": "GenerateFlowFile", + "type": "org.apache.nifi.processors.standard.GenerateFlowFile", + "schedulingStrategy": "TIMER_DRIVEN", + "schedulingPeriod": "10 sec" + }], + "outputPorts": [{ + "identifier": "00000000-0000-0000-0000-000000000003", + "name": "out" + }], + "connections": [{ + "identifier": "00000000-0000-0000-0000-000000000004", + "name": "GenerateFlowFileToOutputPort", + "source": {"id": "00000000-0000-0000-0000-000000000002"}, + "destination": {"id": "00000000-0000-0000-0000-000000000003"}, + "selectedRelationships": ["success"] + }] + }] + } +})"; + + REQUIRE_THROWS_WITH(config.getRootFromPayload(CONFIG_JSON), unterminatedRelationshipsError("[(out, success)]")); +} + +TEST_CASE("NiFi flow json connection with both empty and named relationships only uses the named relationships", "[RelationshipTermination]") { + ConfigurationTestController test_controller; + core::flow::AdaptiveConfiguration config(test_controller.getContext()); + + static const std::string CONFIG_JSON = + R"( +{ + "rootGroup": { + "name": "MiNiFi Flow", + "processors": [{ + "identifier": "00000000-0000-0000-0000-000000000001", + "name": "GenerateFlowFile", + "type": "org.apache.nifi.processors.standard.GenerateFlowFile", + "schedulingStrategy": "TIMER_DRIVEN", + "schedulingPeriod": "10 sec" + }, { + "identifier": "00000000-0000-0000-0000-000000000002", + "name": "LogAttribute", + "type": "org.apache.nifi.processors.standard.LogAttribute", + "schedulingStrategy": "EVENT_DRIVEN", + "autoTerminatedRelationships": ["success"] + }], + "connections": [{ + "identifier": "00000000-0000-0000-0000-000000000003", + "name": "GenerateFlowFileToLogAttribute", + "source": {"id": "00000000-0000-0000-0000-000000000001"}, + "destination": {"id": "00000000-0000-0000-0000-000000000002"}, + "selectedRelationships": ["", "success"] + }] + } +})"; + + std::unique_ptr flow; + REQUIRE_NOTHROW(flow = config.getRootFromPayload(CONFIG_JSON)); + REQUIRE(flow); + + std::map connections; + flow->getConnections(connections); + CHECK(connections.at("GenerateFlowFileToLogAttribute")->getRelationships() == (std::set{{"success", ""}})); +} + } // namespace org::apache::nifi::minifi::test diff --git a/extensions/standard-processors/tests/unit/YamlConfigurationTests.cpp b/extensions/standard-processors/tests/unit/YamlConfigurationTests.cpp index 1f815d3bde..0ba68ff2d3 100644 --- a/extensions/standard-processors/tests/unit/YamlConfigurationTests.cpp +++ b/extensions/standard-processors/tests/unit/YamlConfigurationTests.cpp @@ -24,6 +24,7 @@ #include "core/RepositoryFactory.h" #include "core/yaml/YamlConfiguration.h" #include "TailFile.h" +#include "Funnel.h" #include "unit/TestBase.h" #include "unit/Catch.h" #include "utils/StringUtils.h" @@ -33,6 +34,7 @@ #include "utils/crypto/property_encryption/PropertyEncryptionUtils.h" #include "unit/DummyProcessor.h" #include "catch2/generators/catch_generators.hpp" +#include "catch2/matchers/catch_matchers_string.hpp" using namespace std::literals::chrono_literals; @@ -506,6 +508,8 @@ Flow Controller: Processors: - name: GenerateFlowFile class: GenerateFlowFile + auto-terminated relationships list: + - success Properties: Dynamic Property: Bad )"; @@ -564,6 +568,8 @@ Flow Controller: Processors: - name: XYZ class: GetFile + auto-terminated relationships list: + - success Properties: Input Directory: "/" Batch Size: 1 @@ -2330,6 +2336,8 @@ Flow Controller: max concurrent tasks: 1 scheduling strategy: TIMER_DRIVEN scheduling period: 10000 ms + auto-terminated relationships list: + - success Properties: File Size: {} )", file_size_property_str); @@ -2357,7 +2365,9 @@ Flow Controller: class: org.apache.nifi.processors.standard.PutFile max concurrent tasks: 1 scheduling strategy: EVENT_DRIVEN - auto-terminated relationships list: [] + auto-terminated relationships list: + - success + - failure Properties: [] Connections: @@ -2397,4 +2407,557 @@ NiFi Properties Overrides: {} CHECK(port.get().getTimeout().value() == std::chrono::seconds(20)); } +namespace { +std::string unterminatedRelationshipsError(std::string_view processor_relationship_pairs) { + return fmt::format("General Operation: The following component-relationship pairs are not terminated by any connection: {}", + processor_relationship_pairs); +} +} // namespace + +TEST_CASE("Flow with all relationships terminated by connections or auto-termination is loaded", "[YamlConfiguration][RelationshipTermination]") { + ConfigurationTestController test_controller; + static const std::string CONFIG_YAML = + R"( +MiNiFi Config Version: 3 +Flow Controller: + name: root +Processors: +- id: 00000000-0000-0000-0000-000000000001 + name: GenerateFlowFile + class: org.apache.nifi.minifi.processors.GenerateFlowFile + scheduling strategy: TIMER_DRIVEN + scheduling period: 10 sec +- id: 00000000-0000-0000-0000-000000000002 + name: PutFile + class: org.apache.nifi.minifi.processors.PutFile + scheduling strategy: EVENT_DRIVEN + auto-terminated relationships list: + - success +- id: 00000000-0000-0000-0000-000000000003 + name: LogAttribute + class: org.apache.nifi.minifi.processors.LogAttribute + scheduling strategy: EVENT_DRIVEN + auto-terminated relationships list: + - success +Connections: +- id: 00000000-0000-0000-0000-000000000004 + name: GenerateFlowFile/success/PutFile + source id: 00000000-0000-0000-0000-000000000001 + source relationship names: + - success + destination id: 00000000-0000-0000-0000-000000000002 +- id: 00000000-0000-0000-0000-000000000005 + name: PutFile/failure/LogAttribute + source id: 00000000-0000-0000-0000-000000000002 + source relationship names: + - failure + destination id: 00000000-0000-0000-0000-000000000003 +)"; + core::YamlConfiguration yaml_config(test_controller.getContext()); + std::unique_ptr root; + REQUIRE_NOTHROW(root = yaml_config.getRootFromPayload(CONFIG_YAML)); + REQUIRE(root); +} + +TEST_CASE("Relationship which is neither connected nor auto-terminated is reported", "[YamlConfiguration][RelationshipTermination]") { + ConfigurationTestController test_controller; + static const std::string CONFIG_YAML = + R"( +MiNiFi Config Version: 3 +Flow Controller: + name: root +Processors: +- id: 00000000-0000-0000-0000-000000000001 + name: GenerateFlowFile + class: org.apache.nifi.minifi.processors.GenerateFlowFile + scheduling strategy: TIMER_DRIVEN + scheduling period: 10 sec +Connections: [] +)"; + core::YamlConfiguration yaml_config(test_controller.getContext()); + REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(CONFIG_YAML), unterminatedRelationshipsError("[(GenerateFlowFile, success)]")); +} + +TEST_CASE("Connection of a different relationship does not terminate a relationship", "[YamlConfiguration][RelationshipTermination]") { + ConfigurationTestController test_controller; + static const std::string CONFIG_YAML = + R"( +MiNiFi Config Version: 3 +Flow Controller: + name: root +Processors: +- id: 00000000-0000-0000-0000-000000000001 + name: GenerateFlowFile + class: org.apache.nifi.minifi.processors.GenerateFlowFile + scheduling strategy: TIMER_DRIVEN + scheduling period: 10 sec +- id: 00000000-0000-0000-0000-000000000002 + name: PutFile + class: org.apache.nifi.minifi.processors.PutFile + scheduling strategy: EVENT_DRIVEN +- id: 00000000-0000-0000-0000-000000000003 + name: LogAttribute + class: org.apache.nifi.minifi.processors.LogAttribute + scheduling strategy: EVENT_DRIVEN + auto-terminated relationships list: + - success +Connections: +- id: 00000000-0000-0000-0000-000000000004 + name: GenerateFlowFile/success/PutFile + source id: 00000000-0000-0000-0000-000000000001 + source relationship names: + - success + destination id: 00000000-0000-0000-0000-000000000002 +- id: 00000000-0000-0000-0000-000000000005 + name: PutFile/success/LogAttribute + source id: 00000000-0000-0000-0000-000000000002 + source relationship names: + - success + destination id: 00000000-0000-0000-0000-000000000003 +)"; + core::YamlConfiguration yaml_config(test_controller.getContext()); + REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(CONFIG_YAML), unterminatedRelationshipsError("[(PutFile, failure)]")); +} + +TEST_CASE("Connection from a different processor does not terminate a relationship", "[YamlConfiguration][RelationshipTermination]") { + ConfigurationTestController test_controller; + static const std::string CONFIG_YAML = + R"( +MiNiFi Config Version: 3 +Flow Controller: + name: root +Processors: +- id: 00000000-0000-0000-0000-000000000001 + name: GenerateFlowFile1 + class: org.apache.nifi.minifi.processors.GenerateFlowFile + scheduling strategy: TIMER_DRIVEN + scheduling period: 10 sec +- id: 00000000-0000-0000-0000-000000000002 + name: GenerateFlowFile2 + class: org.apache.nifi.minifi.processors.GenerateFlowFile + scheduling strategy: TIMER_DRIVEN + scheduling period: 10 sec +- id: 00000000-0000-0000-0000-000000000003 + name: LogAttribute + class: org.apache.nifi.minifi.processors.LogAttribute + scheduling strategy: EVENT_DRIVEN + auto-terminated relationships list: + - success +Connections: +- id: 00000000-0000-0000-0000-000000000004 + name: GenerateFlowFile1/success/LogAttribute + source id: 00000000-0000-0000-0000-000000000001 + source relationship names: + - success + destination id: 00000000-0000-0000-0000-000000000003 +)"; + core::YamlConfiguration yaml_config(test_controller.getContext()); + REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(CONFIG_YAML), unterminatedRelationshipsError("[(GenerateFlowFile2, success)]")); +} + +TEST_CASE("All unterminated relationships of all processors are reported", "[YamlConfiguration][RelationshipTermination]") { + ConfigurationTestController test_controller; + static const std::string CONFIG_YAML = + R"( +MiNiFi Config Version: 3 +Flow Controller: + name: root +Processors: +- id: 00000000-0000-0000-0000-000000000001 + name: GenerateFlowFile1 + class: org.apache.nifi.minifi.processors.GenerateFlowFile + scheduling strategy: TIMER_DRIVEN + scheduling period: 10 sec +- id: 00000000-0000-0000-0000-000000000002 + name: GenerateFlowFile2 + class: org.apache.nifi.minifi.processors.GenerateFlowFile + scheduling strategy: TIMER_DRIVEN + scheduling period: 10 sec +- id: 00000000-0000-0000-0000-000000000003 + name: PutFile + class: org.apache.nifi.minifi.processors.PutFile + scheduling strategy: EVENT_DRIVEN +Connections: +- id: 00000000-0000-0000-0000-000000000004 + name: GenerateFlowFile1/success/PutFile + source id: 00000000-0000-0000-0000-000000000001 + source relationship names: + - success + destination id: 00000000-0000-0000-0000-000000000003 +)"; + core::YamlConfiguration yaml_config(test_controller.getContext()); + REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(CONFIG_YAML), + Catch::Matchers::ContainsSubstring("The following component-relationship pairs are not terminated by any connection:") && + Catch::Matchers::ContainsSubstring("(PutFile, failure)(PutFile, success)") && + Catch::Matchers::ContainsSubstring("(GenerateFlowFile2, success)") && !Catch::Matchers::ContainsSubstring("GenerateFlowFile1")); +} + +TEST_CASE("Funnel with or without outgoing connection is accepted", "[YamlConfiguration][RelationshipTermination]") { + ConfigurationTestController test_controller; + static const std::string CONFIG_YAML = + R"( +MiNiFi Config Version: 3 +Flow Controller: + name: root +Processors: +- id: 00000000-0000-0000-0000-000000000001 + name: GenerateFlowFile + class: org.apache.nifi.minifi.processors.GenerateFlowFile + scheduling strategy: TIMER_DRIVEN + scheduling period: 10 sec +- id: 00000000-0000-0000-0000-000000000002 + name: LogAttribute + class: org.apache.nifi.minifi.processors.LogAttribute + scheduling strategy: EVENT_DRIVEN + auto-terminated relationships list: + - success +Funnels: +- id: 00000000-0000-0000-0000-000000000003 + name: MyFunnel +Connections: +- id: 00000000-0000-0000-0000-000000000004 + name: GenerateFlowFile/success/MyFunnel + source id: 00000000-0000-0000-0000-000000000001 + source relationship names: + - success + destination id: 00000000-0000-0000-0000-000000000003 +)"; + static const std::string FUNNEL_OUTGOING_CONNECTION_YAML = + R"( +- id: 00000000-0000-0000-0000-000000000005 + name: MyFunnel//LogAttribute + source id: 00000000-0000-0000-0000-000000000003 + source relationship names: [] + destination id: 00000000-0000-0000-0000-000000000002 +)"; + + std::unique_ptr root; + + SECTION("Funnel is connected") { + core::YamlConfiguration yaml_config(test_controller.getContext()); + REQUIRE_NOTHROW(root = yaml_config.getRootFromPayload(CONFIG_YAML + FUNNEL_OUTGOING_CONNECTION_YAML)); + } + + SECTION("Funnel is not connected, it is used as a terminator") { + static const std::string LOG_ATTRIBUTE_INCOMING_CONNECTION_YAML = + R"( +- id: 00000000-0000-0000-0000-000000000005 + name: GenerateFlowFile/success/LogAttribute + source id: 00000000-0000-0000-0000-000000000001 + source relationship names: + - success + destination id: 00000000-0000-0000-0000-000000000002 +)"; + core::YamlConfiguration yaml_config(test_controller.getContext()); + REQUIRE_NOTHROW(root = yaml_config.getRootFromPayload(CONFIG_YAML + LOG_ATTRIBUTE_INCOMING_CONNECTION_YAML)); + } + + REQUIRE(root); + auto* funnel = root->findProcessorByName("MyFunnel"); + REQUIRE(funnel); + CHECK(funnel->isAutoTerminated(minifi::Funnel::Success)); +} + +class ProcessorWithDefaultAutoTerminatedRelationship : public minifi::core::ProcessorImpl { + public: + using ProcessorImpl::ProcessorImpl; + + static constexpr const char* Description = "test resource"; + static constexpr auto Properties = std::array{}; + static constexpr auto Success = core::RelationshipDefinition{"success", "Success relationship"}; + static constexpr auto Failure = core::RelationshipDefinition{"failure", "Failure relationship"}; + static constexpr auto Original = core::RelationshipDefinition{"original", "Auto-terminated by default"}; + static constexpr auto Relationships = std::array{Success, Failure, Original}; + static constexpr bool SupportsDynamicProperties = false; + static constexpr bool SupportsDynamicRelationships = false; + static constexpr core::annotation::Input InputRequirement = core::annotation::Input::INPUT_ALLOWED; + static constexpr bool IsSingleThreaded = false; + + ADD_COMMON_VIRTUAL_FUNCTIONS_FOR_PROCESSORS + + void initialize() override { + setSupportedRelationships(Relationships); + addAutoTerminatedRelationship(Original); + } +}; + +REGISTER_RESOURCE(ProcessorWithDefaultAutoTerminatedRelationship, Processor); + +TEST_CASE("Relationships auto-terminated by default by the processor do not need to be terminated in the flow configuration", "[YamlConfiguration][RelationshipTermination]") { + ConfigurationTestController test_controller; + core::YamlConfiguration yaml_config(test_controller.getContext()); + + static const std::string CONFIG_YAML = + R"( +MiNiFi Config Version: 3 +Flow Controller: + name: root +Processors: +- id: 00000000-0000-0000-0000-000000000001 + name: MyProcessor + class: ProcessorWithDefaultAutoTerminatedRelationship + auto-terminated relationships list: + - success +)"; + + SECTION("The default auto-terminated relationship is kept besides the ones in the flow configuration") { + std::unique_ptr root; + REQUIRE_NOTHROW(root = yaml_config.getRootFromPayload(CONFIG_YAML + " - failure\n")); + REQUIRE(root); + auto* processor = root->findProcessorByName("MyProcessor"); + REQUIRE(processor); + CHECK(processor->isAutoTerminated(ProcessorWithDefaultAutoTerminatedRelationship::Success)); + CHECK(processor->isAutoTerminated(ProcessorWithDefaultAutoTerminatedRelationship::Failure)); + CHECK(processor->isAutoTerminated(ProcessorWithDefaultAutoTerminatedRelationship::Original)); + } + + SECTION("Other relationships are still verified") { + REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(CONFIG_YAML), unterminatedRelationshipsError("[(MyProcessor, failure)]")); + } +} + +TEST_CASE("Remote process group input ports do not need outgoing connections", "[YamlConfiguration][RelationshipTermination]") { + ConfigurationTestController test_controller; + static const std::string CONFIG_YAML = + R"( +MiNiFi Config Version: 3 +Flow Controller: + name: root +Processors: +- id: 00000000-0000-0000-0000-000000000001 + name: GenerateFlowFile + class: org.apache.nifi.minifi.processors.GenerateFlowFile + scheduling strategy: TIMER_DRIVEN + scheduling period: 10 sec +Connections: +- id: 00000000-0000-0000-0000-000000000002 + name: GenerateFlowFile/success/AmazingInputPort + source id: 00000000-0000-0000-0000-000000000001 + source relationship names: + - success + destination id: 00000000-0000-0000-0000-000000000004 +Remote Process Groups: +- id: 00000000-0000-0000-0000-000000000003 + name: RPG + url: http://localhost:8080/nifi + transport protocol: RAW + Input Ports: + - id: 00000000-0000-0000-0000-000000000004 + name: AmazingInputPort + max concurrent tasks: 1 + Output Ports: [] +)"; + core::YamlConfiguration yaml_config(test_controller.getContext()); + REQUIRE_NOTHROW(yaml_config.getRootFromPayload(CONFIG_YAML)); +} + +TEST_CASE("Remote process group output ports need outgoing connections", "[YamlConfiguration][RelationshipTermination]") { + ConfigurationTestController test_controller; + static const std::string CONFIG_YAML = + R"( +MiNiFi Config Version: 3 +Flow Controller: + name: root +Processors: +- id: 00000000-0000-0000-0000-000000000001 + name: GenerateFlowFile + class: org.apache.nifi.minifi.processors.GenerateFlowFile + scheduling strategy: TIMER_DRIVEN + scheduling period: 10 sec +- id: 00000000-0000-0000-0000-000000000002 + name: LogAttribute + class: org.apache.nifi.minifi.processors.LogAttribute + scheduling strategy: EVENT_DRIVEN + auto-terminated relationships list: + - success +Remote Process Groups: +- id: 00000000-0000-0000-0000-000000000003 + name: RPG + url: http://localhost:8080/nifi + transport protocol: RAW + Input Ports: [] + Output Ports: + - id: 00000000-0000-0000-0000-000000000004 + name: AmazingOutputPort + max concurrent tasks: 1 +Connections: +)"; + + SECTION("Output port is connected") { + static const std::string CONNECTIONS_YAML = + R"( +- id: 00000000-0000-0000-0000-000000000005 + name: AmazingOutputPort/undefined/LogAttribute + source id: 00000000-0000-0000-0000-000000000004 + destination id: 00000000-0000-0000-0000-000000000002 +- id: 00000000-0000-0000-0000-000000000006 + name: GenerateFlowFile/success/LogAttribute + source id: 00000000-0000-0000-0000-000000000001 + source relationship names: + - success + destination id: 00000000-0000-0000-0000-000000000002 +)"; + core::YamlConfiguration yaml_config(test_controller.getContext()); + REQUIRE_NOTHROW(yaml_config.getRootFromPayload(CONFIG_YAML + CONNECTIONS_YAML)); + } + + SECTION("Output port is not connected") { + static const std::string CONNECTIONS_YAML = + R"( +- id: 00000000-0000-0000-0000-000000000006 + name: GenerateFlowFile/success/LogAttribute + source id: 00000000-0000-0000-0000-000000000001 + source relationship names: + - success + destination id: 00000000-0000-0000-0000-000000000002 +)"; + core::YamlConfiguration yaml_config(test_controller.getContext()); + REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(CONFIG_YAML + CONNECTIONS_YAML), + unterminatedRelationshipsError("[(AmazingOutputPort, undefined)]")); + } +} + +TEST_CASE("Relationships are verified in nested process groups", "[YamlConfiguration][RelationshipTermination]") { + ConfigurationTestController test_controller; + static const std::string CONFIG_YAML_BEGIN = + R"( +MiNiFi Config Version: 3 +Flow Controller: + name: root +Processors: +- id: 00000000-0000-0000-0000-000000000001 + name: GenerateFlowFile + class: org.apache.nifi.minifi.processors.GenerateFlowFile + scheduling strategy: TIMER_DRIVEN + scheduling period: 10 sec +- id: 00000000-0000-0000-0000-000000000002 + name: LogAttribute + class: org.apache.nifi.minifi.processors.LogAttribute + scheduling strategy: EVENT_DRIVEN + auto-terminated relationships list: + - success +Connections: +- id: 00000000-0000-0000-0000-000000000003 + name: GenerateFlowFile/success/in + source id: 00000000-0000-0000-0000-000000000001 + source relationship names: + - success + destination id: 00000000-0000-0000-0000-000000000005 +- id: 00000000-0000-0000-0000-000000000004 + name: out/success/LogAttribute + source id: 00000000-0000-0000-0000-000000000006 + source relationship names: + - success + destination id: 00000000-0000-0000-0000-000000000002 +Process Groups: +- id: 00000000-0000-0000-0000-000000000010 + name: Child Process Group + Processors: + - id: 00000000-0000-0000-0000-000000000007 + name: UpdateAttribute + class: org.apache.nifi.minifi.processors.UpdateAttribute + scheduling strategy: EVENT_DRIVEN + auto-terminated relationships list: + - failure + Input Ports: + - id: 00000000-0000-0000-0000-000000000005 + name: in + Output Ports: + - id: 00000000-0000-0000-0000-000000000006 + name: out + Connections: + - id: 00000000-0000-0000-0000-000000000008 + name: in/success/UpdateAttribute + source id: 00000000-0000-0000-0000-000000000005 + source relationship names: + - success + destination id: 00000000-0000-0000-0000-000000000007 +)"; + + SECTION("All relationships are terminated") { + static const std::string CONFIG_YAML_END = + R"( + - id: 00000000-0000-0000-0000-000000000009 + name: UpdateAttribute/success/out + source id: 00000000-0000-0000-0000-000000000007 + source relationship names: + - success + destination id: 00000000-0000-0000-0000-000000000006 +)"; + core::YamlConfiguration yaml_config(test_controller.getContext()); + REQUIRE_NOTHROW(yaml_config.getRootFromPayload(CONFIG_YAML_BEGIN + CONFIG_YAML_END)); + } + + SECTION("Relationship of a processor in the child process group is not terminated") { + core::YamlConfiguration yaml_config(test_controller.getContext()); + REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(CONFIG_YAML_BEGIN), unterminatedRelationshipsError("[(UpdateAttribute, success)]")); + } +} + +TEST_CASE("Process group port without outgoing connection is reported", "[YamlConfiguration][RelationshipTermination]") { + ConfigurationTestController test_controller; + + SECTION("Input port") { + static const std::string CONFIG_YAML = + R"( +MiNiFi Config Version: 3 +Flow Controller: + name: root +Processors: +- id: 00000000-0000-0000-0000-000000000001 + name: GenerateFlowFile + class: org.apache.nifi.minifi.processors.GenerateFlowFile + scheduling strategy: TIMER_DRIVEN + scheduling period: 10 sec +Connections: +- id: 00000000-0000-0000-0000-000000000002 + name: GenerateFlowFile/success/in + source id: 00000000-0000-0000-0000-000000000001 + source relationship names: + - success + destination id: 00000000-0000-0000-0000-000000000004 +Process Groups: +- id: 00000000-0000-0000-0000-000000000003 + name: Child Process Group + Processors: [] + Input Ports: + - id: 00000000-0000-0000-0000-000000000004 + name: in +)"; + core::YamlConfiguration yaml_config(test_controller.getContext()); + REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(CONFIG_YAML), unterminatedRelationshipsError("[(in, success)]")); + } + + SECTION("Output port") { + static const std::string CONFIG_YAML = + R"( +MiNiFi Config Version: 3 +Flow Controller: + name: root +Processors: [] +Process Groups: +- id: 00000000-0000-0000-0000-000000000001 + name: Child Process Group + Processors: + - id: 00000000-0000-0000-0000-000000000002 + name: GenerateFlowFile + class: org.apache.nifi.minifi.processors.GenerateFlowFile + scheduling strategy: TIMER_DRIVEN + scheduling period: 10 sec + Output Ports: + - id: 00000000-0000-0000-0000-000000000003 + name: out + Connections: + - id: 00000000-0000-0000-0000-000000000004 + name: GenerateFlowFile/success/out + source id: 00000000-0000-0000-0000-000000000002 + source relationship names: + - success + destination id: 00000000-0000-0000-0000-000000000003 +)"; + core::YamlConfiguration yaml_config(test_controller.getContext()); + REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(CONFIG_YAML), unterminatedRelationshipsError("[(out, success)]")); + } +} + } // namespace org::apache::nifi::minifi::test diff --git a/extensions/standard-processors/tests/unit/YamlProcessGroupParserTests.cpp b/extensions/standard-processors/tests/unit/YamlProcessGroupParserTests.cpp index 2e1dac807c..bec7415e18 100644 --- a/extensions/standard-processors/tests/unit/YamlProcessGroupParserTests.cpp +++ b/extensions/standard-processors/tests/unit/YamlProcessGroupParserTests.cpp @@ -145,6 +145,10 @@ TEST_CASE("Processor can communicate with child process group's input port", "[Y .With({ Group("Child1") .With({InputPort{"00000000-0000-0000-0000-000000000002", "Port1"}}) + .With({Proc{"00000000-0000-0000-0000-000000000003", "Child1_Proc1"}}) + .With({Conn{"Child1_Conn1", + InputPort{"00000000-0000-0000-0000-000000000002", "Port1"}, + Proc{"00000000-0000-0000-0000-000000000003", "Child1_Proc1"}}}) }); auto root = config.getRootFromPayload(pattern.serialize().join("\n")); @@ -179,6 +183,10 @@ TEST_CASE("Child process groups can communicate through ports", "[YamlProcessGro .With({OutputPort{"00000000-0000-0000-0000-000000000002", "Port1"}}), Group("Child2") .With({InputPort{"00000000-0000-0000-0000-000000000003", "Port2"}}) + .With({Proc{"00000000-0000-0000-0000-000000000004", "Child2_Proc1"}}) + .With({Conn{"Child2_Conn1", + InputPort{"00000000-0000-0000-0000-000000000003", "Port2"}, + Proc{"00000000-0000-0000-0000-000000000004", "Child2_Proc1"}}}) }); auto root = config.getRootFromPayload(pattern.serialize().join("\n")); @@ -190,17 +198,25 @@ TEST_CASE("Processor cannot communicate with child's nested process group", "[Ya Proc Proc1{"00000000-0000-0000-0000-000000000001", "Proc1"}; OutputPort Port1{"00000000-0000-0000-0000-000000000002", "Port1"}; InputPort Port2{"00000000-0000-0000-0000-000000000003", "Port2", ConnectionFailure::UNRESOLVED_DESTINATION}; + Proc Child2_Proc1{"00000000-0000-0000-0000-000000000004", "Child2_Proc1"}; auto pattern = Group("root") .With({Conn{"Conn1", Proc1, - Port2}}) + Port2}, + Conn{"Conn2", + Port1, + Proc1}}) .With({Proc1}) .With({ Group("Child1") .With({Port1}) .With({Group("Child2") - .With({Port2})}) + .With({Port2}) + .With({Child2_Proc1}) + .With({Conn{"Child2_Conn1", + InputPort{Port2.id, Port2.name}, + Child2_Proc1}})}) }); auto root = config.getRootFromPayload(pattern.serialize().join("\n")); @@ -210,11 +226,18 @@ TEST_CASE("Processor cannot communicate with child's nested process group", "[Ya TEST_CASE("Input port can be a connection's source and the output port can be a destination inside the process group", "[YamlProcessGroupParser7]") { auto pattern = Group("root") - .With({Conn{"Conn1", - InputPort{"00000000-0000-0000-0000-000000000001", "Port1"}, - OutputPort{"00000000-0000-0000-0000-000000000002", "Port2"}}}) - .With({InputPort{"00000000-0000-0000-0000-000000000001", "Port1"}}) - .With({OutputPort{"00000000-0000-0000-0000-000000000002", "Port2"}}); + .With({Conn{"Conn2", + OutputPort{"00000000-0000-0000-0000-000000000002", "Port2"}, + Proc{"00000000-0000-0000-0000-000000000003", "Proc1"}}}) + .With({Proc{"00000000-0000-0000-0000-000000000003", "Proc1"}}) + .With({ + Group("Child1") + .With({Conn{"Conn1", + InputPort{"00000000-0000-0000-0000-000000000001", "Port1"}, + OutputPort{"00000000-0000-0000-0000-000000000002", "Port2"}}}) + .With({InputPort{"00000000-0000-0000-0000-000000000001", "Port1"}}) + .With({OutputPort{"00000000-0000-0000-0000-000000000002", "Port2"}}) + }); auto root = config.getRootFromPayload(pattern.serialize().join("\n")); @@ -225,7 +248,10 @@ TEST_CASE("Input port cannot be a connection's destination inside the process gr auto pattern = Group("root") .With({Conn{"Conn1", Proc{"00000000-0000-0000-0000-000000000002", "Proc1"}, - InputPort{"00000000-0000-0000-0000-000000000001", "Port1", ConnectionFailure::INPUT_CANNOT_BE_DESTINATION}}}) + InputPort{"00000000-0000-0000-0000-000000000001", "Port1", ConnectionFailure::INPUT_CANNOT_BE_DESTINATION}}, + Conn{"Conn2", + InputPort{"00000000-0000-0000-0000-000000000001", "Port1"}, + Proc{"00000000-0000-0000-0000-000000000002", "Proc1"}}}) .With({InputPort{"00000000-0000-0000-0000-000000000001", "Port1"}}) .With({Proc{"00000000-0000-0000-0000-000000000002", "Proc1"}}); @@ -235,12 +261,20 @@ TEST_CASE("Input port cannot be a connection's destination inside the process gr } TEST_CASE("Output port cannot be a connection's source inside the process group", "[YamlProcessGroupParser]") { + // the output port is in a child process group, because the output port can only have an outgoing connection in the parent process group auto pattern = Group("root") - .With({Conn{"Conn1", - OutputPort{"00000000-0000-0000-0000-000000000001", "Port1", ConnectionFailure::OUTPUT_CANNOT_BE_SOURCE}, - Proc{"00000000-0000-0000-0000-000000000002", "Proc1"}}}) - .With({OutputPort{"00000000-0000-0000-0000-000000000001", "Port1"}}) - .With({Proc{"00000000-0000-0000-0000-000000000002", "Proc1"}}); + .With({Conn{"Conn2", + OutputPort{"00000000-0000-0000-0000-000000000001", "Port1"}, + Proc{"00000000-0000-0000-0000-000000000003", "Proc2"}}}) + .With({Proc{"00000000-0000-0000-0000-000000000003", "Proc2"}}) + .With({ + Group("Child1") + .With({Conn{"Conn1", + OutputPort{"00000000-0000-0000-0000-000000000001", "Port1", ConnectionFailure::OUTPUT_CANNOT_BE_SOURCE}, + Proc{"00000000-0000-0000-0000-000000000002", "Proc1"}}}) + .With({OutputPort{"00000000-0000-0000-0000-000000000001", "Port1"}}) + .With({Proc{"00000000-0000-0000-0000-000000000002", "Proc1"}}) + }); auto root = config.getRootFromPayload(pattern.serialize().join("\n")); @@ -249,15 +283,22 @@ TEST_CASE("Output port cannot be a connection's source inside the process group" TEST_CASE("Input port can be a connection's source and the output port can be a destination inside the process group through processor", "[YamlProcessGroupParser]") { auto pattern = Group("root") - .With({Conn{"Conn1", - InputPort{"00000000-0000-0000-0000-000000000001", "Port1"}, - Proc{"00000000-0000-0000-0000-000000000003", "Proc1"}}, - Conn{"Conn2", - Proc{"00000000-0000-0000-0000-000000000003", "Proc1"}, - OutputPort{"00000000-0000-0000-0000-000000000002", "Port2"}}}) - .With({Proc{"00000000-0000-0000-0000-000000000003", "Proc1"}}) - .With({InputPort{"00000000-0000-0000-0000-000000000001", "Port1"}}) - .With({OutputPort{"00000000-0000-0000-0000-000000000002", "Port2"}}); + .With({Conn{"Conn3", + OutputPort{"00000000-0000-0000-0000-000000000002", "Port2"}, + Proc{"00000000-0000-0000-0000-000000000004", "Proc2"}}}) + .With({Proc{"00000000-0000-0000-0000-000000000004", "Proc2"}}) + .With({ + Group("Child1") + .With({Conn{"Conn1", + InputPort{"00000000-0000-0000-0000-000000000001", "Port1"}, + Proc{"00000000-0000-0000-0000-000000000003", "Proc1"}}, + Conn{"Conn2", + Proc{"00000000-0000-0000-0000-000000000003", "Proc1"}, + OutputPort{"00000000-0000-0000-0000-000000000002", "Port2"}}}) + .With({Proc{"00000000-0000-0000-0000-000000000003", "Proc1"}}) + .With({InputPort{"00000000-0000-0000-0000-000000000001", "Port1"}}) + .With({OutputPort{"00000000-0000-0000-0000-000000000002", "Port2"}}) + }); auto root = config.getRootFromPayload(pattern.serialize().join("\n")); @@ -268,7 +309,10 @@ TEST_CASE("Processor cannot set connection's destination to child process group' auto pattern = Group("root") .With({Conn{"Conn1", Proc{"00000000-0000-0000-0000-000000000001", "Proc1"}, - OutputPort{"00000000-0000-0000-0000-000000000002", "Port1", ConnectionFailure::OUTPUT_CANNOT_BE_DESTINATION}}}) + OutputPort{"00000000-0000-0000-0000-000000000002", "Port1", ConnectionFailure::OUTPUT_CANNOT_BE_DESTINATION}}, + Conn{"Conn2", + OutputPort{"00000000-0000-0000-0000-000000000002", "Port1"}, + Proc{"00000000-0000-0000-0000-000000000001", "Proc1"}}}) .With({Proc{"00000000-0000-0000-0000-000000000001", "Proc1"}}) .With({ Group("Child1") @@ -289,6 +333,10 @@ TEST_CASE("Processor cannot set connection's source to child process group's inp .With({ Group("Child1") .With({InputPort{"00000000-0000-0000-0000-000000000002", "Port1"}}) + .With({Proc{"00000000-0000-0000-0000-000000000003", "Child1_Proc1"}}) + .With({Conn{"Child1_Conn1", + InputPort{"00000000-0000-0000-0000-000000000002", "Port1"}, + Proc{"00000000-0000-0000-0000-000000000003", "Child1_Proc1"}}}) }); auto root = config.getRootFromPayload(pattern.serialize().join("\n")); diff --git a/libminifi/include/Funnel.h b/libminifi/include/Funnel.h index 51ad1fa12a..e2d00a826c 100644 --- a/libminifi/include/Funnel.h +++ b/libminifi/include/Funnel.h @@ -32,7 +32,7 @@ class Funnel final : public ForwardingNode { MINIFIAPI static constexpr core::annotation::Input InputRequirement = core::annotation::Input::INPUT_REQUIRED; ADD_COMMON_VIRTUAL_FUNCTIONS_FOR_PROCESSORS - void onSchedule(core::ProcessContext&, core::ProcessSessionFactory&) override; + void initialize() override; }; } // namespace org::apache::nifi::minifi diff --git a/libminifi/include/RemoteProcessGroupPort.h b/libminifi/include/RemoteProcessGroupPort.h index b1da2d90d1..8b7023a47e 100644 --- a/libminifi/include/RemoteProcessGroupPort.h +++ b/libminifi/include/RemoteProcessGroupPort.h @@ -110,6 +110,10 @@ class RemoteProcessGroupPort : public core::ProcessorImpl { transmitting_ = val; } + [[nodiscard]] sitetosite::TransferDirection getDirection() const { + return direction_; + } + void setInterface(const std::string& ifc) { local_network_interface_ = ifc; } diff --git a/libminifi/include/core/ProcessGroup.h b/libminifi/include/core/ProcessGroup.h index 06a3ef4dfa..4b6d20f917 100644 --- a/libminifi/include/core/ProcessGroup.h +++ b/libminifi/include/core/ProcessGroup.h @@ -189,9 +189,9 @@ class ProcessGroup : public CoreComponentImpl { void updatePropertyValue(const std::string& processorName, const std::string& propertyName, const std::string& propertyValue); - void getConnections(std::map& connectionMap); + void getConnections(std::map& connectionMap) const; - void getConnections(std::map& connectionMap); + void getConnections(std::map& connectionMap) const; void getFlowFileContainers(std::map& containers) const; @@ -232,6 +232,7 @@ class ProcessGroup : public CoreComponentImpl { private: static Port* findPortById(const std::set& ports, const utils::Identifier& uuid); std::string buildGroupPath() const; + void verifyRelationshipTerminations() const; mutable std::recursive_mutex mutex_; std::shared_ptr logger_; diff --git a/libminifi/src/Funnel.cpp b/libminifi/src/Funnel.cpp index a3ef769f3f..97b18d8627 100644 --- a/libminifi/src/Funnel.cpp +++ b/libminifi/src/Funnel.cpp @@ -16,12 +16,12 @@ */ #include "Funnel.h" -#include "minifi-cpp/core/ProcessContext.h" namespace org::apache::nifi::minifi { -void Funnel::onSchedule(core::ProcessContext& context, core::ProcessSessionFactory&) { - context.addAutoTerminatedRelationship(Success); +void Funnel::initialize() { + ForwardingNode::initialize(); + addAutoTerminatedRelationship(Success); } } // namespace org::apache::nifi::minifi diff --git a/libminifi/src/core/ProcessGroup.cpp b/libminifi/src/core/ProcessGroup.cpp index 36d5e89466..510757c0d1 100644 --- a/libminifi/src/core/ProcessGroup.cpp +++ b/libminifi/src/core/ProcessGroup.cpp @@ -23,11 +23,13 @@ #include #include #include +#include #include #include #include "core/Processor.h" #include "core/state/ProcessorController.h" #include "core/state/UpdateController.h" +#include "RemoteProcessGroupPort.h" using namespace std::literals::chrono_literals; @@ -349,7 +351,7 @@ void ProcessGroup::updatePropertyValue(const std::string& processorName, const s } } -void ProcessGroup::getConnections(std::map& connectionMap) { +void ProcessGroup::getConnections(std::map& connectionMap) const { for (auto& connection : connections_) { connectionMap[connection->getUUIDStr()] = connection.get(); connectionMap[connection->getName()] = connection.get(); @@ -359,7 +361,7 @@ void ProcessGroup::getConnections(std::map& connection } } -void ProcessGroup::getConnections(std::map& connectionMap) { +void ProcessGroup::getConnections(std::map& connectionMap) const { for (auto& connection : connections_) { connectionMap[connection->getUUIDStr()] = connection.get(); connectionMap[connection->getName()] = connection.get(); @@ -494,10 +496,52 @@ std::size_t ProcessGroup::getTotalFlowFileCount() const { return sum; } +void ProcessGroup::verifyRelationshipTerminations() const { + std::map connections; + getConnections(connections); + std::set> terminated_relationships; + for (const auto& [_, connection] : connections) { + for (const auto& relationship : connection->getRelationships()) { + terminated_relationships.emplace(connection->getSource(), relationship.getName()); + } + } + + std::vector processors; + getAllProcessors(processors); + + std::vector> unterminated_relationships; + for (auto* processor : processors) { + // RPG input ports (SEND direction) only send their incoming flow files to the remote instance and never transfer to a local relationship, + // while RPG output ports (RECEIVE direction) transfer the received flow files to their relationship, so those are still verified + if (auto port = dynamic_cast(&processor->getImpl()); + port && port->getDirection() == sitetosite::TransferDirection::SEND) { + continue; + } + for (const auto& relationship : processor->getSupportedRelationships()) { + if (processor->isAutoTerminated(relationship)) { + continue; + } + if (!terminated_relationships.contains({processor, relationship.getName()})) { + unterminated_relationships.emplace_back(processor->getName(), relationship.getName()); + } + } + } + + if (!unterminated_relationships.empty()) { + std::string message = "The following component-relationship pairs are not terminated by any connection: ["; + for (const auto& [processor, relationship] : unterminated_relationships) { + message.append(fmt::format("({}, {})", processor, relationship)); + } + message += "]"; + throw Exception(ExceptionType::GENERAL_EXCEPTION, message); + } +} + void ProcessGroup::verify() const { for (const auto& processor : processors_) { processor->validateAnnotations(); } + verifyRelationshipTerminations(); } void ProcessGroup::setParameterContext(ParameterContext* parameter_context) { diff --git a/libminifi/src/core/Processor.cpp b/libminifi/src/core/Processor.cpp index b20004a092..4b24de4aa9 100644 --- a/libminifi/src/core/Processor.cpp +++ b/libminifi/src/core/Processor.cpp @@ -364,6 +364,10 @@ class ProcessorDescriptorImpl : public ProcessorDescriptor { impl_->setSupportedProperties(properties); } + void addAutoTerminatedRelationship(const RelationshipDefinition& relationship) override { + impl_->addAutoTerminatedRelationship(relationship); + } + private: Processor* impl_; }; diff --git a/libminifi/src/core/flow/StructuredConfiguration.cpp b/libminifi/src/core/flow/StructuredConfiguration.cpp index 47324cb62b..8cf9273900 100644 --- a/libminifi/src/core/flow/StructuredConfiguration.cpp +++ b/libminifi/src/core/flow/StructuredConfiguration.cpp @@ -444,15 +444,12 @@ void StructuredConfiguration::parseProcessorNode(const Node& processors_node, co processor->setRunDurationNano(std::chrono::nanoseconds(*run_duration_nanos)); } - std::vector autoTerminatedRelationships; for (auto &&relString : procCfg.autoTerminatedRelationships) { core::Relationship relationship(relString, ""); logger_->log_debug("parseProcessorNode: autoTerminatedRelationship => [{}]", relString); - autoTerminatedRelationships.push_back(relationship); + processor->addAutoTerminatedRelationship(relationship); } - processor->setAutoTerminatedRelationships(autoTerminatedRelationships); - parentGroup->addProcessor(std::move(processor)); } } @@ -1002,6 +999,7 @@ void StructuredConfiguration::parseFunnels(const Node& node, core::ProcessGroup* }); auto funnel = std::make_unique("Funnel", name, uuid.value(), std::make_unique(name, uuid.value())); + funnel->initialize(); logger_->log_debug("Created funnel with UUID {} and name {}", id, name); funnel->setScheduledState(core::RUNNING); funnel->setSchedulingStrategy(core::EVENT_DRIVEN); @@ -1030,6 +1028,7 @@ void StructuredConfiguration::parsePorts(const flow::Node& node, core::ProcessGr }); auto port = std::make_unique(name, uuid.value(), std::make_unique(name, uuid.value(), port_type)); + port->initialize(); logger_->log_debug("Created port UUID {} and name {}", id, name); port->setScheduledState(core::RUNNING); port->setSchedulingStrategy(core::EVENT_DRIVEN); diff --git a/libminifi/src/core/flow/StructuredConnectionParser.cpp b/libminifi/src/core/flow/StructuredConnectionParser.cpp index fdfbcd58ce..7b0a210c2d 100644 --- a/libminifi/src/core/flow/StructuredConnectionParser.cpp +++ b/libminifi/src/core/flow/StructuredConnectionParser.cpp @@ -19,6 +19,7 @@ #include "core/flow/StructuredConnectionParser.h" #include "core/flow/CheckRequiredField.h" #include "Funnel.h" +#include "Port.h" #include "RemoteProcessGroupPort.h" namespace org::apache::nifi::minifi::core::flow { @@ -47,6 +48,8 @@ void StructuredConnectionParser::addFunnelOrPortRelationshipToConnection(minifi: addNewRelationshipToConnection(minifi::Funnel::Success.name, connection); } else if (typeid(minifi::RemoteProcessGroupPort) == typeid(impl)) { addNewRelationshipToConnection(minifi::RemoteProcessGroupPort::DefaultRelationship.name, connection); + } else if (typeid(minifi::PortImpl) == typeid(impl)) { + addNewRelationshipToConnection(minifi::PortImpl::Success.name, connection); } else { logger_->log_error("Component with id {} is neither funnel nor port", srcUUID.to_string()); } @@ -58,8 +61,18 @@ void StructuredConnectionParser::configureConnectionSourceRelationships(minifi:: addNewRelationshipToConnection(connectionNode_[schema_.source_relationship].getString().value(), connection); } else if (auto relList = connectionNode_[schema_.source_relationship_list]) { if (relList.isSequence() && !relList.empty()) { + // NiFi serializes the anonymous relationship of funnels and ports as an empty string: "selectedRelationships": [""] + bool has_named_relationship = false; for (const auto &rel : relList) { - addNewRelationshipToConnection(rel.getString().value(), connection); + const auto relationship_name = rel.getString().value(); + if (relationship_name.empty()) { + continue; + } + addNewRelationshipToConnection(relationship_name, connection); + has_named_relationship = true; + } + if (!has_named_relationship) { + addFunnelOrPortRelationshipToConnection(connection); } } else if (!relList.isSequence() && !relList.getString().value().empty()) { addNewRelationshipToConnection(relList.getString().value(), connection); diff --git a/libminifi/test/flow-tests/CycleTest.cpp b/libminifi/test/flow-tests/CycleTest.cpp index d4c62a69e2..567ff3e7b9 100644 --- a/libminifi/test/flow-tests/CycleTest.cpp +++ b/libminifi/test/flow-tests/CycleTest.cpp @@ -48,6 +48,7 @@ Flow Controller: yield period: 100 ms run duration nanos: 0 auto-terminated relationships list: + - banana Properties: AppleProbability: 100 BananaProbability: 0 @@ -61,6 +62,7 @@ Flow Controller: yield period: 100 ms run duration nanos: 0 auto-terminated relationships list: + - banana Properties: AppleProbability: 100 BananaProbability: 0 diff --git a/libminifi/test/flow-tests/FlowControllerTests.cpp b/libminifi/test/flow-tests/FlowControllerTests.cpp index 2c0a7d4957..085f15b696 100644 --- a/libminifi/test/flow-tests/FlowControllerTests.cpp +++ b/libminifi/test/flow-tests/FlowControllerTests.cpp @@ -32,6 +32,7 @@ #include "unit/TestControllerWithFlow.h" #include "unit/EmptyFlow.h" #include "unit/TestUtils.h" +#include "catch2/matchers/catch_matchers_string.hpp" using namespace std::literals::chrono_literals; @@ -261,6 +262,8 @@ Flow Controller: class: GenerateFlowFile scheduling strategy: TIMER_DRIVEN scheduling period: 10 sec + auto-terminated relationships list: + - success Connections: [] Remote Processing Groups: [] Controller Services: [] @@ -284,3 +287,119 @@ Controller Services: [] REQUIRE(minifi::test::utils::verifyLogLinePresenceInPollTime(0s, "Destroying FlowController")); REQUIRE(LogTestController::getInstance().countOccurrences("Destroying scheduling agent") == 3); } + +TEST_CASE("Flow update with unterminated relationships is rejected and the current flow is kept", "[TestFlow6]") { + TestControllerWithFlow controller(R"( +Flow Controller: + name: Banana Bread +Processors: +- name: GenFF + id: 00000000-0000-0000-0000-000000000001 + class: GenerateFlowFile + scheduling strategy: TIMER_DRIVEN + scheduling period: 10 sec + auto-terminated relationships list: + - success +Connections: [] +Remote Processing Groups: [] +Controller Services: [] +)"); + + controller.startFlow(); + + const auto result = controller.controller_->applyConfiguration("/flows/1", R"( +Flow Controller: + name: Unterminated Flow +Processors: +- name: UnterminatedGenFF + id: 00000000-0000-0000-0000-000000000002 + class: GenerateFlowFile + scheduling strategy: TIMER_DRIVEN + scheduling period: 10 sec +Connections: [] +Remote Processing Groups: [] +Controller Services: [] +)"); + + REQUIRE_FALSE(result.has_value()); + CHECK_THAT(result.error(), Catch::Matchers::ContainsSubstring("not terminated by any connection: [(UnterminatedGenFF, success)]")); + CHECK(controller.controller_->getName() == "Banana Bread"); + CHECK(controller.root_->findProcessorByName("GenFF")); + CHECK_FALSE(controller.root_->findProcessorByName("UnterminatedGenFF")); +} + +TEST_CASE("Flow files are routed through process group ports", "[TestFlow7]") { + // the connections from the ports have no relationship set, so they use the default relationship of the port + TestControllerWithFlow controller(R"( +Flow Controller: + name: Port Routing Flow +Processors: +- name: Generator + id: 00000000-0000-0000-0000-000000000001 + class: org.apache.nifi.processors.standard.TestFlowFileGenerator + scheduling strategy: TIMER_DRIVEN + scheduling period: 100 ms +- name: Sink + id: 00000000-0000-0000-0000-000000000002 + class: org.apache.nifi.processors.standard.TestProcessor + scheduling strategy: TIMER_DRIVEN + scheduling period: 100 ms + auto-terminated relationships list: + - apple + - banana +Connections: +- name: GeneratorToChildInputPort + id: 00000000-0000-0000-0000-000000000003 + source id: 00000000-0000-0000-0000-000000000001 + source relationship names: + - success + destination id: 00000000-0000-0000-0000-000000000011 +- name: ChildOutputPortToSink + id: 00000000-0000-0000-0000-000000000004 + source id: 00000000-0000-0000-0000-000000000012 + destination id: 00000000-0000-0000-0000-000000000002 +Process Groups: +- name: Child + id: 00000000-0000-0000-0000-000000000010 + Processors: + - name: Middle + id: 00000000-0000-0000-0000-000000000013 + class: org.apache.nifi.processors.standard.TestProcessor + scheduling strategy: TIMER_DRIVEN + scheduling period: 100 ms + auto-terminated relationships list: + - banana + Input Ports: + - id: 00000000-0000-0000-0000-000000000011 + name: in + Output Ports: + - id: 00000000-0000-0000-0000-000000000012 + name: out + Connections: + - name: ChildInputPortToMiddle + id: 00000000-0000-0000-0000-000000000014 + source id: 00000000-0000-0000-0000-000000000011 + destination id: 00000000-0000-0000-0000-000000000013 + - name: MiddleToChildOutputPort + id: 00000000-0000-0000-0000-000000000015 + source id: 00000000-0000-0000-0000-000000000013 + source relationship names: + - apple + destination id: 00000000-0000-0000-0000-000000000012 +Remote Processing Groups: [] +Controller Services: [] +)"); + + TypedProcessorWrapper sink_proc = controller.root_->findProcessorByName("Sink"); + REQUIRE(sink_proc); + // prevent the sink from consuming the flow files, so they stay in its incoming connection + sink_proc->yield(10s); + + std::map connections; + controller.root_->getConnections(connections); + auto* const sink_connection = connections.at("ChildOutputPortToSink"); + + controller.startFlow(); + + CHECK(minifi::test::utils::verifyEventHappenedInPollTime(5s, [&] { return sink_connection->getQueueSize() > 0; })); +} diff --git a/libminifi/test/flow-tests/LoopTest.cpp b/libminifi/test/flow-tests/LoopTest.cpp index e78d84fe3c..12d80f7234 100644 --- a/libminifi/test/flow-tests/LoopTest.cpp +++ b/libminifi/test/flow-tests/LoopTest.cpp @@ -47,6 +47,7 @@ Flow Controller: yield period: 100 ms run duration nanos: 0 auto-terminated relationships list: + - banana Properties: AppleProbability: 100 BananaProbability: 0 diff --git a/libminifi/test/integration/HTTPSiteToSiteTests.cpp b/libminifi/test/integration/HTTPSiteToSiteTests.cpp index faa487bdd3..a83b63a53a 100644 --- a/libminifi/test/integration/HTTPSiteToSiteTests.cpp +++ b/libminifi/test/integration/HTTPSiteToSiteTests.cpp @@ -87,8 +87,6 @@ struct test_profile { }; void run_variance(const std::filesystem::path& test_file_location, const std::string& url, const struct test_profile &profile) { - SiteToSiteTestHarness harness(test_file_location); - std::string in_port = "471deef6-2a6e-4a7d-912a-81cc17e3a204"; std::string out_port = "471deef6-2a6e-4a7d-912a-81cc17e3a203"; @@ -99,31 +97,21 @@ void run_variance(const std::filesystem::path& test_file_location, const std::st std::string transaction_id = transaction_response->getTransactionId(); - harness.setKeyDir(""); - std::string controller_loc = url + "/controller"; std::string basesitetosite = url + "/site-to-site"; auto base = std::make_unique(basesitetosite); - harness.setUrl(basesitetosite, base.get()); - - harness.setUrl(controller_loc, responder.get()); - std::string transaction_url = url + "/data-transfer/input-ports/" + in_port + "/transactions"; std::string action_url = url + "/site-to-site/input-ports/" + in_port + "/transactions"; std::string transaction_output_url = url + "/data-transfer/output-ports/" + out_port + "/transactions"; std::string action_output_url = url + "/site-to-site/output-ports/" + out_port + "/transactions"; - harness.setUrl(transaction_url, transaction_response.get()); - std::string peer_url = url + "/site-to-site/peers"; auto peer_response = std::make_unique(url); - harness.setUrl(peer_url, peer_response.get()); - std::string flow_url = action_url + "/" + transaction_id + "/flow-files"; auto flowResponder = std::make_unique(true, profile.flow_url_broken, profile.invalid_checksum); @@ -135,23 +123,31 @@ void run_variance(const std::filesystem::path& test_file_location, const std::st std::string transaction_output_id = transaction_response_output->getTransactionId(); transaction_response_output->setFeed(producedFlows); - harness.setUrl(transaction_output_url, transaction_response_output.get()); - std::string flow_output_url = action_output_url + "/" + transaction_output_id + "/flow-files"; auto flowOutputResponder = std::make_unique(false, profile.flow_url_broken, profile.invalid_checksum); flowOutputResponder->setFlowUrl(flow_output_url); flowOutputResponder->setFeed(producedFlows); - harness.setUrl(flow_url, flowResponder.get()); - harness.setUrl(flow_output_url, flowOutputResponder.get()); - std::string delete_url = transaction_url + "/" + transaction_id; auto deleteResponse = std::make_unique(delete_url, "201 OK", 12); - harness.setUrl(delete_url, deleteResponse.get()); std::string delete_output_url = transaction_output_url + "/" + transaction_output_id; auto deleteOutputResponse = std::make_unique(delete_output_url, "201 OK", producedFlows); + + // the harness has to be destroyed before the responders, because its test server accesses the handlers on shutdown + SiteToSiteTestHarness harness(test_file_location); + + harness.setKeyDir(""); + + harness.setUrl(basesitetosite, base.get()); + harness.setUrl(controller_loc, responder.get()); + harness.setUrl(transaction_url, transaction_response.get()); + harness.setUrl(peer_url, peer_response.get()); + harness.setUrl(transaction_output_url, transaction_response_output.get()); + harness.setUrl(flow_url, flowResponder.get()); + harness.setUrl(flow_output_url, flowOutputResponder.get()); + harness.setUrl(delete_url, deleteResponse.get()); harness.setUrl(delete_output_url, deleteOutputResponse.get()); harness.run(); diff --git a/libminifi/test/resources/C2VerifyHeartbeatAndStop.yml b/libminifi/test/resources/C2VerifyHeartbeatAndStop.yml index 5211eb2288..6c5a99d94c 100644 --- a/libminifi/test/resources/C2VerifyHeartbeatAndStop.yml +++ b/libminifi/test/resources/C2VerifyHeartbeatAndStop.yml @@ -30,6 +30,10 @@ Processors: yield period: 1 sec run duration nanos: 0 auto-terminated relationships list: + - failure + - no retry + - response + - retry Properties: HTTP Method: GET Remote URL: http://localhost:10015/geturl @@ -43,7 +47,6 @@ Processors: yield period: 1 sec run duration nanos: 0 auto-terminated relationships list: - - response Properties: Log Level: info Log Payload: true @@ -64,11 +67,11 @@ Connections: source name: LogAttribute source id: 2438e3c8-015a-1000-79ca-83af40ec1992 destination name: LogAttribute - destination id: 2438e3c8-015a-1000-79ca-83af40ec1992 + destination id: 2438e3c8-015a-1000-79ca-83af40ec1992 source relationship name: success max work queue size: 0 max work queue data size: 1 MB flowfile expiration: 60 sec Remote Processing Groups: - + diff --git a/libminifi/test/resources/C2VerifyHeartbeatAndStopSecure.yml b/libminifi/test/resources/C2VerifyHeartbeatAndStopSecure.yml index 125db3640b..681af10e3d 100644 --- a/libminifi/test/resources/C2VerifyHeartbeatAndStopSecure.yml +++ b/libminifi/test/resources/C2VerifyHeartbeatAndStopSecure.yml @@ -30,6 +30,10 @@ Processors: yield period: 1 sec run duration nanos: 0 auto-terminated relationships list: + - failure + - no retry + - response + - retry Properties: HTTP Method: GET Remote URL: https://localhost:11015/geturl @@ -42,8 +46,6 @@ Processors: penalization period: 30 sec yield period: 1 sec run duration nanos: 0 - auto-terminated relationships list: - - response Properties: Log Level: info Log Payload: true @@ -64,11 +66,11 @@ Connections: source name: LogAttribute source id: 2438e3c8-015a-1000-79ca-83af40ec1992 destination name: LogAttribute - destination id: 2438e3c8-015a-1000-79ca-83af40ec1992 + destination id: 2438e3c8-015a-1000-79ca-83af40ec1992 source relationship name: success max work queue size: 0 max work queue data size: 1 MB flowfile expiration: 60 sec Remote Processing Groups: - + diff --git a/libminifi/test/resources/TestC2Metrics.yml b/libminifi/test/resources/TestC2Metrics.yml index fdb8924198..9171706aec 100644 --- a/libminifi/test/resources/TestC2Metrics.yml +++ b/libminifi/test/resources/TestC2Metrics.yml @@ -31,6 +31,7 @@ Processors: run duration nanos: 0 bulletin level: ERROR auto-terminated relationships list: + - partial Properties: Endpoint List: localhost:8776 Message Delimiter: \r @@ -47,7 +48,7 @@ Processors: bulletin level: ERROR run duration nanos: 0 auto-terminated relationships list: - - response + - success Properties: Log Level: info Log Payload: true diff --git a/libminifi/test/resources/TestC2MetricsUpdate.yml b/libminifi/test/resources/TestC2MetricsUpdate.yml index 477402a536..f188cdc999 100644 --- a/libminifi/test/resources/TestC2MetricsUpdate.yml +++ b/libminifi/test/resources/TestC2MetricsUpdate.yml @@ -45,7 +45,7 @@ Processors: yield period: 1 sec run duration nanos: 0 auto-terminated relationships list: - - response + - success Properties: Log Level: info Log Payload: true diff --git a/libminifi/test/resources/TestHTTPPost.yml b/libminifi/test/resources/TestHTTPPost.yml index 080d4e670c..7ef2bd532b 100644 --- a/libminifi/test/resources/TestHTTPPost.yml +++ b/libminifi/test/resources/TestHTTPPost.yml @@ -58,6 +58,7 @@ Processors: - no retry - failure - response + - success Properties: HTTP Method: POST Content-type: text/html @@ -76,7 +77,7 @@ Processors: Properties: Log Level: debug -Connections: +Connections: - name: Generate/Loggit id: 2438e3c8-015a-1000-79ca-83af40ec1919 source id: 2438e3c8-015a-1000-79ca-83af40ec1991 @@ -105,4 +106,4 @@ Connections: max work queue data size: 1 MB flowfile expiration: 60 sec Remote Processing Groups: - + diff --git a/libminifi/test/resources/TestHTTPSiteToSite.yml b/libminifi/test/resources/TestHTTPSiteToSite.yml index 619cbd5ac5..2081d7f09f 100644 --- a/libminifi/test/resources/TestHTTPSiteToSite.yml +++ b/libminifi/test/resources/TestHTTPSiteToSite.yml @@ -41,6 +41,7 @@ Processors: yield period: 10 sec run duration nanos: 0 auto-terminated relationships list: + - success Properties: @@ -57,7 +58,6 @@ Connections: - name: GenerateFlowFileS2S id: 471deef6-2a6e-4a7d-912a-81cc17e3a210 source id: 471deef6-2a6e-4a7d-912a-81cc17e3a203 - source relationship name: success destination id: 471deef6-2a6e-4a7d-912a-81cc17e3a205 max work queue size: 0 max work queue data size: 1 MB diff --git a/libminifi/test/resources/TestNull.yml b/libminifi/test/resources/TestNull.yml index 5211eb2288..9e8c0dcfb1 100644 --- a/libminifi/test/resources/TestNull.yml +++ b/libminifi/test/resources/TestNull.yml @@ -30,6 +30,10 @@ Processors: yield period: 1 sec run duration nanos: 0 auto-terminated relationships list: + - failure + - no retry + - response + - retry Properties: HTTP Method: GET Remote URL: http://localhost:10015/geturl @@ -43,7 +47,7 @@ Processors: yield period: 1 sec run duration nanos: 0 auto-terminated relationships list: - - response + - success Properties: Log Level: info Log Payload: true @@ -64,11 +68,10 @@ Connections: source name: LogAttribute source id: 2438e3c8-015a-1000-79ca-83af40ec1992 destination name: LogAttribute - destination id: 2438e3c8-015a-1000-79ca-83af40ec1992 + destination id: 2438e3c8-015a-1000-79ca-83af40ec1992 source relationship name: success max work queue size: 0 max work queue data size: 1 MB flowfile expiration: 60 sec Remote Processing Groups: - diff --git a/libminifi/test/resources/TestUpdateAttribute.yml b/libminifi/test/resources/TestUpdateAttribute.yml index 522484e5e2..7f4b6cbe68 100644 --- a/libminifi/test/resources/TestUpdateAttribute.yml +++ b/libminifi/test/resources/TestUpdateAttribute.yml @@ -39,6 +39,8 @@ Processors: penalization period: 30 sec yield period: 1 sec run duration nanos: 0 + auto-terminated relationships list: + - failure Properties: route_condition_attr: true - name: roa @@ -49,8 +51,11 @@ Processors: penalization period: 30 sec yield period: 1 sec run duration nanos: 0 + auto-terminated relationships list: + - failure + - unmatched Properties: - route_matched: ${route_condition_attr} + route_matched: ${route_condition_attr} - name: up2 id: 2438e3c8-015a-1000-79ca-83af40ec1994 class: org.apache.nifi.processors.standard.UpdateAttribute @@ -59,6 +64,8 @@ Processors: penalization period: 30 sec yield period: 1 sec run duration nanos: 0 + auto-terminated relationships list: + - failure Properties: route_check_attr: good variable_attribute: ${nifi.variable.test} @@ -114,6 +121,6 @@ Connections: source relationship name: success max work queue size: 0 max work queue data size: 1 MB - flowfile expiration: 60 sec + flowfile expiration: 60 sec Remote Processing Groups: - + diff --git a/libminifi/test/resources/decrypted.config.json b/libminifi/test/resources/decrypted.config.json index 0089dc4d96..3ef3ce4647 100644 --- a/libminifi/test/resources/decrypted.config.json +++ b/libminifi/test/resources/decrypted.config.json @@ -74,7 +74,7 @@ "bulletinLevel": "WARN", "runDurationMillis": 0, "concurrentlySchedulableTaskCount": 1, - "autoTerminatedRelationships": [], + "autoTerminatedRelationships": ["success"], "componentType": "PROCESSOR", "groupIdentifier": "9d7b1592-5d7b-4630-8677-2208e48c5d14" } diff --git a/libminifi/test/resources/decrypted.config.json.reformatted b/libminifi/test/resources/decrypted.config.json.reformatted index 0089dc4d96..0684ef8604 100644 --- a/libminifi/test/resources/decrypted.config.json.reformatted +++ b/libminifi/test/resources/decrypted.config.json.reformatted @@ -74,7 +74,9 @@ "bulletinLevel": "WARN", "runDurationMillis": 0, "concurrentlySchedulableTaskCount": 1, - "autoTerminatedRelationships": [], + "autoTerminatedRelationships": [ + "success" + ], "componentType": "PROCESSOR", "groupIdentifier": "9d7b1592-5d7b-4630-8677-2208e48c5d14" } diff --git a/minifi-api/include/minifi-cpp/core/ProcessorDescriptor.h b/minifi-api/include/minifi-cpp/core/ProcessorDescriptor.h index 9ebc2a3843..371b7bddd9 100644 --- a/minifi-api/include/minifi-cpp/core/ProcessorDescriptor.h +++ b/minifi-api/include/minifi-cpp/core/ProcessorDescriptor.h @@ -29,6 +29,7 @@ class ProcessorDescriptor { virtual void setSupportedRelationships(std::span relationships) = 0; virtual void setSupportedProperties(std::span properties) = 0; virtual void setSupportedProperties(std::span properties) = 0; + virtual void addAutoTerminatedRelationship(const RelationshipDefinition& relationship) = 0; }; } // namespace org::apache::nifi::minifi::core diff --git a/minifi_rust/extensions/minifi_pgp/features/encrypt_decrypt.feature b/minifi_rust/extensions/minifi_pgp/features/encrypt_decrypt.feature index 47c7d48c82..73b86ff306 100644 --- a/minifi_rust/extensions/minifi_pgp/features/encrypt_decrypt.feature +++ b/minifi_rust/extensions/minifi_pgp/features/encrypt_decrypt.feature @@ -57,7 +57,12 @@ Feature: Test PGP extension's encryption and decryption capabilities | DecryptBob | failure | BobFailure | And AliceSuccess's success relationship is auto-terminated + And AliceSuccess's failure relationship is auto-terminated And BobFailure's success relationship is auto-terminated + And BobFailure's failure relationship is auto-terminated + And EncryptContentPGP's failure relationship is auto-terminated + And DecryptAlice's failure relationship is auto-terminated + And DecryptBob's success relationship is auto-terminated And a directory at "/tmp/input" has a file "test_file.log" with the content "test content" diff --git a/minifi_rust/extensions/minifi_rs_playground/features/basic.feature b/minifi_rust/extensions/minifi_rs_playground/features/basic.feature index 698266740d..9cc14234a5 100644 --- a/minifi_rust/extensions/minifi_rs_playground/features/basic.feature +++ b/minifi_rust/extensions/minifi_rs_playground/features/basic.feature @@ -37,6 +37,7 @@ Feature: Basic scenarios And a PutFileRs processor with the "Directory" property set to "/tmp/output" And the "success" relationship of the GenerateFlowFileRs processor is connected to the PutFileRs And PutFileRs's success relationship is auto-terminated + And PutFileRs's failure relationship is auto-terminated When the MiNiFi instance starts up @@ -87,6 +88,7 @@ Feature: Basic scenarios And a PutFileRs processor with the "Directory" property set to "/tmp/output" And the "success" relationship of the LoremIpsumCSUser processor is connected to the PutFileRs And PutFileRs's success relationship is auto-terminated + And PutFileRs's failure relationship is auto-terminated When the MiNiFi instance starts up diff --git a/minifi_rust/extensions/minifi_rs_playground/features/streaming.feature b/minifi_rust/extensions/minifi_rs_playground/features/streaming.feature index 2189156516..4389461eba 100644 --- a/minifi_rust/extensions/minifi_rs_playground/features/streaming.feature +++ b/minifi_rust/extensions/minifi_rs_playground/features/streaming.feature @@ -22,6 +22,7 @@ Feature: Testing streaming reads and writes And a PutFileRs processor with the "Directory" property set to "/tmp/output" And the "success" relationship of the GetFileRs processor is connected to the AsciifyGerman And the "success" relationship of the AsciifyGerman processor is connected to the PutFileRs + And AsciifyGerman's failure relationship is auto-terminated And PutFileRs's success relationship is auto-terminated And PutFileRs's failure relationship is auto-terminated And a directory at "/tmp/input" has a file "german.txt" with the content "Üben von Xylophon und Querflöte ist ja zweckmäßig." @@ -38,6 +39,7 @@ Feature: Testing streaming reads and writes And a PutFileRs processor with the "Directory" property set to "/tmp/output" And the "success" relationship of the GetFileRs processor is connected to the AsciifyGerman And the "failure" relationship of the AsciifyGerman processor is connected to the PutFileRs + And AsciifyGerman's success relationship is auto-terminated And PutFileRs's success relationship is auto-terminated And PutFileRs's failure relationship is auto-terminated And a directory at "/tmp/input" has a file "french.txt" with the content "Voix ambiguë d'un cœur qui, au zéphyr, préfère les jattes de kiwis."