MINIFICPP-2935 Verify that all relationships are terminated in the flow configuration

CONFIGURE.md

@@ -142,6 +142,7 @@ Remote Processing Groups:
 Besides YAML configuration format, MiNiFi C++ also supports JSON configuration. To see different uses cases in both formats, please refer to the [examples page](examples/README.md) for flow config examples.
 
 **NOTE:** Make sure to specify id for each component (Processor, Connection, Controller, RPG etc.) to make sure that Apache MiNiFi C++ can reload the state after a process restart. The id should be unique in the flow configuration.
+**NOTE:** Make sure to terminate all outgoing connections of processors, ports, funnels, either by connecting them to another component or auto-terminating them, otherwise the flow validation fails on startup.
 
 ### Parameter Contexts
 

core-framework/include/core/ProcessorImpl.h

@@ -84,6 +84,8 @@ class ProcessorImpl : public virtual ProcessorApi {
   void setSupportedProperties(std::span<const PropertyReference> properties);
   void setSupportedProperties(std::span<const Property> properties);
 
+  void addAutoTerminatedRelationship(const RelationshipDefinition& relationship);
+
   virtual void initialize() {}
 
   void onTrigger(ProcessContext&, ProcessSession&) override {}

core-framework/src/core/ProcessorImpl.cpp

@@ -84,6 +84,11 @@ void ProcessorImpl::setSupportedRelationships(std::span<const RelationshipDefini
   descriptor_->setSupportedRelationships(relationships);
 }
 
+void ProcessorImpl::addAutoTerminatedRelationship(const RelationshipDefinition& relationship) {
+  gsl_Expects(descriptor_);
+  descriptor_->addAutoTerminatedRelationship(relationship);
+}
+
 void ProcessorImpl::setSupportedProperties(std::span<const PropertyReference> properties) {
   gsl_Expects(descriptor_);
   descriptor_->setSupportedProperties(properties);

examples/couchbase_mtls_authentication.json

@@ -69,6 +69,6 @@
                     "success"
                 ]
             }
-        ],
+        ]
     }
 }

examples/process_data_with_scripts.json

@@ -17,7 +17,8 @@
                     "Script File": "../../examples/scripts/python/reverse_flow_file_content.py"
                 },
                 "autoTerminatedRelationships": [
-                    "failure"
+                    "failure",
+                    "original"
                 ]
             },
             {

examples/process_data_with_scripts.yml

@@ -28,6 +28,7 @@ Processors:
     run duration nanos: 0
     auto-terminated relationships list:
       - failure
+      - original
     Properties:
       Module Directory:
       Script Body:

extensions/aws/tests/features/kinesis.feature

@@ -38,6 +38,7 @@ Feature: Sending data from MiNiFi-C++ to an AWS Kinesis server
     And the "success" relationship of the PutKinesisStream processor is connected to the PutFile
     And the "failure" relationship of the PutKinesisStream processor is connected to the PutKinesisStream
     And PutFile's success relationship is auto-terminated
+    And PutFile's failure relationship is auto-terminated
 
     When both instances start up
 

extensions/aws/tests/features/s3.feature

@@ -28,6 +28,7 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
     And the "success" relationship of the PutS3Object processor is connected to the PutFile
     And the "failure" relationship of the PutS3Object processor is connected to the PutS3Object
     And PutFile's success relationship is auto-terminated
+    And PutFile's failure relationship is auto-terminated
 
     And the s3 server starts up
 
@@ -53,6 +54,7 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
     And the "success" relationship of the PutS3Object processor is connected to the PutFile
     And the "failure" relationship of the PutS3Object processor is connected to the PutS3Object
     And PutFile's success relationship is auto-terminated
+    And PutFile's failure relationship is auto-terminated
 
     And the s3 server starts up
     And the http proxy server is set up
@@ -72,7 +74,9 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
     And the "success" relationship of the GetFile processor is connected to the PutS3Object
     And the "success" relationship of the PutS3Object processor is connected to the PutFile
     And a ProxyConfigurationService controller service is set up with HTTP proxy configuration
+    And PutS3Object's failure relationship is auto-terminated
     And PutFile's success relationship is auto-terminated
+    And PutFile's failure relationship is auto-terminated
 
     And the s3 server starts up
     And the http proxy server is set up
@@ -95,7 +99,9 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
       | PutS3Object    | success           | DeleteS3Object   |
       | PutS3Object    | failure           | PutS3Object      |
       | DeleteS3Object | success           | PutFile          |
+      | DeleteS3Object | failure           | auto-terminated  |
       | PutFile        | success           | auto-terminated  |
+      | PutFile        | failure           | auto-terminated  |
 
     And the s3 server starts up
 
@@ -111,6 +117,9 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
     And a PutFile processor with the "Directory" property set to "/tmp/output"
     And the "success" relationship of the GetFile processor is connected to the DeleteS3Object
     And the "success" relationship of the DeleteS3Object processor is connected to the PutFile
+    And DeleteS3Object's failure relationship is auto-terminated
+    And PutFile's success relationship is auto-terminated
+    And PutFile's failure relationship is auto-terminated
 
     And the s3 server starts up
 
@@ -139,7 +148,9 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
       | PutS3Object    | failure           | PutS3Object      |
       | PutS3Object    | success           | DeleteS3Object   |
       | DeleteS3Object | success           | PutFile          |
+      | DeleteS3Object | failure           | auto-terminated  |
       | PutFile        | success           | auto-terminated  |
+      | PutFile        | failure           | auto-terminated  |
 
     And the s3 server starts up
     And the http proxy server is set up
@@ -162,9 +173,13 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
       | source name    | relationship name | destination name |
       | GetFile        | success           | PutS3Object      |
       | PutS3Object    | success           | DeleteS3Object   |
+      | PutS3Object    | failure           | auto-terminated  |
       | DeleteS3Object | success           | PutFile          |
+      | DeleteS3Object | failure           | auto-terminated  |
+      | PutFile        | success           | auto-terminated  |
+      | PutFile        | failure           | auto-terminated  |
+
     And a ProxyConfigurationService controller service is set up with HTTP proxy configuration
-    And PutFile's success relationship is auto-terminated
 
     And the s3 server starts up
     And the http proxy server is set up
@@ -186,6 +201,7 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
       | source name      | relationship name | destination name |
       | GenerateFlowFile | success           | FetchS3Object    |
       | FetchS3Object    | success           | PutFile          |
+      | FetchS3Object    | failure           | auto-terminated  |
       | PutFile          | success           | auto-terminated  |
       | PutFile          | failure           | auto-terminated  |
       | GetFile          | success           | PutS3Object      |
@@ -218,6 +234,11 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
       | source name      | relationship name | destination name |
       | GenerateFlowFile | success           | FetchS3Object    |
       | FetchS3Object    | success           | PutFile          |
+      | FetchS3Object    | failure           | auto-terminated  |
+      | PutFile          | failure           | auto-terminated  |
+      | PutFile          | success           | auto-terminated  |
+      | PutS3Object      | success           | auto-terminated  |
+      | PutS3Object      | failure           | auto-terminated  |
 
     And the s3 server starts up
     And the http proxy server is set up
@@ -233,6 +254,7 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
     And a PutS3Object processor set up to communicate with an s3 server
     And the "success" relationship of the GetFile processor is connected to the PutS3Object
     And PutS3Object's success relationship is auto-terminated
+    And PutS3Object's failure relationship is auto-terminated
 
     Given a GenerateFlowFile processor with the "File Size" property set to "1 kB"
     And a FetchS3Object processor set up to communicate with the same s3 server
@@ -242,8 +264,10 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
       | source name      | relationship name | destination name |
       | GenerateFlowFile | success           | FetchS3Object    |
       | FetchS3Object    | success           | PutFile          |
+      | FetchS3Object    | failure           | auto-terminated  |
+      | PutFile          | success           | auto-terminated  |
+      | PutFile          | failure           | auto-terminated  |
     And a ProxyConfigurationService controller service is set up with HTTP proxy configuration
-    And PutFile's success relationship is auto-terminated
 
     And the s3 server starts up
     And the http proxy server is set up
@@ -266,6 +290,10 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
     Given a ListS3 processor set up to communicate with the same s3 server
     And a PutFile processor with the "Directory" property set to "/tmp/output"
     And the "success" relationship of the ListS3 processor is connected to the PutFile
+    And PutFile's success relationship is auto-terminated
+    And PutFile's failure relationship is auto-terminated
+    And PutS3Object's success relationship is auto-terminated
+    And PutS3Object's failure relationship is auto-terminated
 
     And the s3 server starts up
 
@@ -278,6 +306,8 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
     And a directory at "/tmp/input" has a file with the content "test"
     And a PutS3Object processor set up to communicate with an s3 server
     And the "success" relationship of the GetFile processor is connected to the PutS3Object
+    And PutS3Object's success relationship is auto-terminated
+    And PutS3Object's failure relationship is auto-terminated
 
     Given a ListS3 processor set up to communicate with the same s3 server
     And these processor properties are set
@@ -288,6 +318,8 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
       | ListS3         | Proxy Password | test101                          |
     And a PutFile processor with the "Directory" property set to "/tmp/output"
     And the "success" relationship of the ListS3 processor is connected to the PutFile
+    And PutFile's success relationship is auto-terminated
+    And PutFile's failure relationship is auto-terminated
 
     And the s3 server starts up
     And the http proxy server is set up
@@ -303,13 +335,15 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
     And a PutS3Object processor set up to communicate with an s3 server
     And the "success" relationship of the GetFile processor is connected to the PutS3Object
     And PutS3Object's success relationship is auto-terminated
+    And PutS3Object's failure relationship is auto-terminated
 
     Given a ListS3 processor set up to communicate with the same s3 server
     And the "Proxy Configuration Service" property of the ListS3 processor is set to "ProxyConfigurationService"
     And a PutFile processor with the "Directory" property set to "/tmp/output"
     And the "success" relationship of the ListS3 processor is connected to the PutFile
     And a ProxyConfigurationService controller service is set up with HTTP proxy configuration
     And PutFile's success relationship is auto-terminated
+    And PutFile's failure relationship is auto-terminated
 
     And the s3 server starts up
     And the http proxy server is set up
@@ -328,6 +362,11 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
     And the "success" relationship of the GetFile processor is connected to the PutS3Object
     And a PutFile processor with the "Directory" property set to "/tmp/output"
     And the "success" relationship of the PutS3Object processor is connected to the PutFile
+    And PutS3Object's success relationship is auto-terminated
+    And PutS3Object's failure relationship is auto-terminated
+    And PutFile's failure relationship is auto-terminated
+    And PutFile's success relationship is auto-terminated
+
     And the s3 server starts up
 
     When the MiNiFi instance starts up
@@ -351,6 +390,9 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
     And the "success" relationship of the GetFile processor is connected to the PutS3Object
     And a PutFile processor with the "Directory" property set to "/tmp/output"
     And the "success" relationship of the PutS3Object processor is connected to the PutFile
+    And PutS3Object's failure relationship is auto-terminated
+    And PutFile's failure relationship is auto-terminated
+    And PutFile's success relationship is auto-terminated
 
     And the s3 server starts up
     And the http proxy server is set up

extensions/azure/tests/features/azure_storage.feature

@@ -28,6 +28,7 @@ Feature: Sending data from MiNiFi-C++ to an Azure storage server
     And the "success" relationship of the PutAzureBlobStorage processor is connected to the PutFile
     And the "failure" relationship of the PutAzureBlobStorage processor is connected to the PutAzureBlobStorage
     And PutFile's success relationship is auto-terminated
+    And PutFile's failure relationship is auto-terminated
 
     And an Azure storage server is set up
     When the MiNiFi instance starts up
@@ -40,6 +41,8 @@ Feature: Sending data from MiNiFi-C++ to an Azure storage server
     And a DeleteAzureBlobStorage processor set up to communicate with an Azure blob storage
     And the "Blob" property of the DeleteAzureBlobStorage processor is set to "test"
     And the "success" relationship of the GenerateFlowFile processor is connected to the DeleteAzureBlobStorage
+    And DeleteAzureBlobStorage's success relationship is auto-terminated
+    And DeleteAzureBlobStorage's failure relationship is auto-terminated
     And an Azure storage server is set up
     And test blob "test" is created on Azure blob storage
 
@@ -54,6 +57,7 @@ Feature: Sending data from MiNiFi-C++ to an Azure storage server
     And the "Delete Snapshots Option" property of the DeleteAzureBlobStorage processor is set to "Include Snapshots"
     And the "success" relationship of the GenerateFlowFile processor is connected to the DeleteAzureBlobStorage
     And DeleteAzureBlobStorage's success relationship is auto-terminated
+    And DeleteAzureBlobStorage's failure relationship is auto-terminated
     And an Azure storage server is set up
     And test blob "test" is created on Azure blob storage with a snapshot
 
@@ -67,6 +71,7 @@ Feature: Sending data from MiNiFi-C++ to an Azure storage server
     And the "Delete Snapshots Option" property of the DeleteAzureBlobStorage processor is set to "Delete Snapshots Only"
     And the "success" relationship of the GenerateFlowFile processor is connected to the DeleteAzureBlobStorage
     And DeleteAzureBlobStorage's success relationship is auto-terminated
+    And DeleteAzureBlobStorage's failure relationship is auto-terminated
     And an Azure storage server is set up
     And test blob "test" is created on Azure blob storage with a snapshot
 
@@ -84,7 +89,9 @@ Feature: Sending data from MiNiFi-C++ to an Azure storage server
     And a PutFile processor with the "Directory" property set to "/tmp/output"
     And the "success" relationship of the GetFile processor is connected to the FetchAzureBlobStorage
     And the "success" relationship of the FetchAzureBlobStorage processor is connected to the PutFile
+    And FetchAzureBlobStorage's failure relationship is auto-terminated
     And PutFile's success relationship is auto-terminated
+    And PutFile's failure relationship is auto-terminated
     And an Azure storage server is set up
 
     When the MiNiFi instance starts up
@@ -119,6 +126,7 @@ Feature: Sending data from MiNiFi-C++ to an Azure storage server
     And the "success" relationship of the PutAzureBlobStorage processor is connected to the PutFile
     And the "failure" relationship of the PutAzureBlobStorage processor is connected to the PutAzureBlobStorage
     And PutFile's success relationship is auto-terminated
+    And PutFile's failure relationship is auto-terminated
     And a ProxyConfigurationService controller service is set up with HTTP proxy configuration
 
     And an Azure storage server is set up
@@ -137,6 +145,7 @@ Feature: Sending data from MiNiFi-C++ to an Azure storage server
     And the "Proxy Configuration Service" property of the DeleteAzureBlobStorage processor is set to "ProxyConfigurationService"
     And the "success" relationship of the GenerateFlowFile processor is connected to the DeleteAzureBlobStorage
     And DeleteAzureBlobStorage's success relationship is auto-terminated
+    And DeleteAzureBlobStorage's failure relationship is auto-terminated
     And a ProxyConfigurationService controller service is set up with HTTP proxy configuration
 
     And an Azure storage server is set up
@@ -161,6 +170,8 @@ Feature: Sending data from MiNiFi-C++ to an Azure storage server
     And the "success" relationship of the GetFile processor is connected to the FetchAzureBlobStorage
     And the "success" relationship of the FetchAzureBlobStorage processor is connected to the PutFile
     And PutFile's success relationship is auto-terminated
+    And PutFile's failure relationship is auto-terminated
+    And FetchAzureBlobStorage's failure relationship is auto-terminated
     And a ProxyConfigurationService controller service is set up with HTTP proxy configuration
 
     And an Azure storage server is set up