• AIPressRoom
  • Posts
  • How I Constructed A Cascading Knowledge Pipeline Based mostly on AWS (Half 2) | by Memphis Meng | Aug, 2023

How I Constructed A Cascading Knowledge Pipeline Based mostly on AWS (Half 2) | by Memphis Meng | Aug, 2023

Computerized, scalable, and highly effective

Previously, I shared my expertise in growing a knowledge pipeline utilizing AWS CloudFormation expertise. It’s not an optimum strategy, although, as a result of it leaves behind 3 extra points awaiting decision:

  1. The deployment must be imposed manually which might improve the possibilities of errors;

  2. All assets are created in a single single stack, with out correct boundaries and layers; as the event cycle goes on, the useful resource stack can be heavier, and managing it will likely be a catastrophe;

  3. Many assets are purported to be sustained and reused in different initiatives.

In brief, we’re going to improve the manageability and reusability of this venture, in an agile method.

AWS permits customers to implement 2 sorts of CloudFormation structural patterns: cross-stack reference and nested stacking. Cross-stack reference stands for a designing fashion of growing cloud stacks individually, and often independently, whereas the assets amongst all stacks will be interrelated based mostly on the reference relationship. Nested stacking means a CloudFormation stack composed of different stacks. It’s achieved by utilizing the AWS::CloudFormation::Stack useful resource.

As a result of one in all our missions we intention to realize is to give you higher venture administration, the venture goes to be damaged down by layered separation and nested stacking is the one to assist. Nonetheless, in regard to the intrinsic interrelationship between the artifacts of the present stack, we might additionally have to take a drop of cross-stack reference.

We created 3 Lambda features, 3 DynamoDB tables, 1 IAM function together with its insurance policies hooked up, a number of SQS queues, and several other Cloudwatch alarms. As a result of complexity of the features themselves, on this model, they’ll be outlined in separate templates, with the companies solely utilized by themselves together with alarms and lifeless letter queues. Other than these, IAM assets can be…