-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
author Gaurav Gaur <Gaur4vGaur@users.noreply.github.com> 1675074395 +0000 committer Gaurav Gaur <Gaur4vGaur@users.noreply.github.com> 1675445316 +0000 parent 6c35647 author Gaurav Gaur <Gaur4vGaur@users.noreply.github.com> 1675074395 +0000 committer Gaurav Gaur <Gaur4vGaur@users.noreply.github.com> 1675445269 +0000 parent 6c35647 author Gaurav Gaur <Gaur4vGaur@users.noreply.github.com> 1675074395 +0000 committer Gaurav Gaur <Gaur4vGaur@users.noreply.github.com> 1675445201 +0000 parent 6c35647 author Gaurav Gaur <Gaur4vGaur@users.noreply.github.com> 1675074395 +0000 committer Gaurav Gaur <Gaur4vGaur@users.noreply.github.com> 1675445170 +0000 parent 6c35647 author Gaurav Gaur <Gaur4vGaur@users.noreply.github.com> 1675074395 +0000 committer Gaurav Gaur <Gaur4vGaur@users.noreply.github.com> 1675445041 +0000 progress with pipes and filters pattern feat: add shimmer background when image loads feat: set preview image ratio to 1.91 : 1 fix: `og:image` will be incorrect if the image uses a cross-domain URL feat: support dark and light mode images (cotes2020#481) refactor: unify the border radius of blocks scope: code blocks, prompts, images, and videos feat: add `rel="me"` to Mastodon sidebar contact links for verification (cotes2020#807) This will enable verification with Mastodon by including the `rel="me"` attribute. https://docs.joinmastodon.org/user/profile/#verification Co-authored-by: Cotes Chung <11371340+cotes2020@users.noreply.github.com> chore: hide blur edge of LQIP images chore(tools): checkout latest tag on initialization chore(editorconfig): keep trailing spaces in Markdown (cotes2020#830) feat(i18n): add Czech language (cotes2020#833) fix: correct encoding of spaces in share URLs (cotes2020#835) Resolves cotes2020#835 fix: post's image would cover the PWA update alert feat(i18n): add Finnish translations (cotes2020#843) chore(bundler): deprecate `:install_if` option in Gemfile feat(i18n): add Italian translation (cotes2020#850) fix: copy command line incomplete(`.gp` part) feat(i18n): add Arabic translation (cotes2020#857) chore(deps): liquid(v4.0.4) is compatible with ruby 3.2 (cotes2020#854) See: jekyll/jekyll#9231 chore: ignore gem lock file in user-end refactor: optimize tab cursor position calculation fix: prompt with nested blockquotes renders incorrectly (cotes2020#846) Resolves cotes2020#846 chore(release): 5.5.0 fix: the icon position of the prompts in the list is incorrect chore(release): 5.5.1
- Loading branch information
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,93 @@ | ||
# The layout text of site | ||
|
||
# ----- Commons label ----- | ||
|
||
layout: | ||
post: منشور | ||
category: فئة | ||
tag: وسم | ||
|
||
# The tabs of sidebar | ||
tabs: | ||
# format: <filename_without_extension>: <value> | ||
home: الرئيسية | ||
categories: الفئات | ||
tags: الوسوم | ||
archives: الأرشيف | ||
about: حول | ||
|
||
# the text displayed in the search bar & search results | ||
search: | ||
hint: بحث | ||
cancel: إلغاء | ||
no_results: نأسف! لا يوجد نتائج. | ||
|
||
panel: | ||
lastmod: المحدثة مؤخرا | ||
trending_tags: الوسوم الشائعة | ||
toc: محتويات | ||
|
||
copyright: | ||
# Shown at the bottom of the post | ||
license: | ||
template: هذا المنشور تحت ترخيص :LICENSE_NAME بواسطة المؤلف. | ||
name: CC BY 4.0 | ||
link: https://creativecommons.org/licenses/by/4.0/ | ||
|
||
# Displayed in the footer | ||
brief: بعض الحقوق محفوظة. | ||
verbose: >- | ||
ما لم يذكر خلاف ذلك ، يتم ترخيص منشورات المدونة على هذا الموقع | ||
بموجب ترخيص Creative Commons Attribution 4.0 International (CC BY 4.0) من قبل المؤلف. | ||
meta: باستخدام :PLATFORM السمة :THEME. | ||
|
||
not_found: | ||
statment: عذرا, الرابط التالي غير صالح أو انه يشير إلى صفحة غير موجودة. | ||
|
||
notification: | ||
update_found: يتوفر اصدار جديد للمحتوى. | ||
update: تحديث | ||
|
||
# ----- Posts related labels ----- | ||
|
||
post: | ||
written_by: بواسطة | ||
posted: نشّر | ||
updated: حدّث | ||
words: كلمات | ||
pageview_measure: مشاهدات | ||
read_time: | ||
unit: دقيقة | ||
prompt: قراءة | ||
relate_posts: إقرأ المزيد | ||
share: شارك | ||
button: | ||
next: الأجدد | ||
previous: الأقدم | ||
copy_code: | ||
succeed: تم النسخ! | ||
share_link: | ||
title: أنسخ الرابط | ||
succeed: تم نسخ الرابط بنجاح! | ||
# pinned prompt of posts list on homepage | ||
pin_prompt: مثبت | ||
|
||
# Date time format. | ||
# See: <http://strftime.net/>, <https://day.js.org/docs/en/display/format> | ||
df: | ||
post: | ||
strftime: '%b %e, %Y' | ||
dayjs: 'll' | ||
archives: | ||
strftime: '%b' | ||
dayjs: 'MMM' | ||
|
||
# categories page | ||
categories: | ||
category_measure: | ||
singular: فئة | ||
plural: فئات | ||
post_measure: | ||
singular: منشور | ||
plural: منشورات |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,5 @@ | ||
/*! | ||
* Chirpy v5.4.0 (https://github.com/cotes2020/jekyll-theme-chirpy/) | ||
* Chirpy v5.5.1 (https://github.com/cotes2020/jekyll-theme-chirpy/) | ||
* © 2019 Cotes Chung | ||
* MIT Licensed | ||
*/ |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,17 +1,46 @@ | ||
--- | ||
title: Pipes And Filters Pattern | ||
description: | ||
tags: ["cloud", "pipes and filter", "design", "scaling"] | ||
tags: ["cloud", "pipes and filters", "design", "scaling"] | ||
category: ["architecture", "patterns"] | ||
date: 2023-01-28 | ||
permalink: '/patterns/pipes-and-filters/' | ||
counterlink: 'patterns-pipes-and-filters/' | ||
image: | ||
path: https://raw.githubusercontent.com/Gaur4vGaur/traveller/master/images/patterns/pipes-and-filters/pipesAndFiltersCover.png | ||
path: https://raw.githubusercontent.com/Gaur4vGaur/traveller/master/images/patterns/2023-01-28-pipes-and-filters-pattern/pipesAndFiltersCover.png | ||
width: 800 | ||
height: 500 | ||
--- | ||
|
||
## Introduction | ||
Applications today collect an infinite amount of data. Many applications need to transform this data before applying any meaningful business logic. Tackling this complex data or a similar processor intensive task without a thought-through scalable strategy can have a high-performance impact. This article introduces another scalability pattern – pipes and filters - appropriate for such scenarios and promote reusability. | ||
Applications today collect an infinite amount of data. Many applications need to transform this data before applying any meaningful business logic. Tackling this complex data or a similar processor intensive task without a thought-through strategy can have a high-performance impact. This article introduces a scalability pattern – pipes and filters – that promotes reusability and is appropriate for such scenarios. | ||
|
||
## Problem Context | ||
Consider a scenario where incoming data triggers a sequence of processing steps, where each step brings data closer to the desired output state. The origin of data is referred to as a data source. Examples of data sources could be home IoT devices, a video feed from roadside cameras, or continuous inventory updates from warehouses. The processing steps during the transformation usually execute a specific operation and are referred to as filters. These processing steps are independent and do not have a side-effect, i.e., running a step does not depend on any other steps. Each filtering step reads the data, performs a transforming operation based on local data and produces an output. Once the data has gone through the filters, it reaches its final processed stage where it is consumed, referred to as Data Sink. | ||
A straightforward implementation could be a complete service that takes the data input, performs all the steps sequentially and produce an output. The modules within the service perform the required step and pass on the data to the next module. Although the solution initially looks good as it hides all the complexity of data processing, the use of such a monolithic service will have listed problems: | ||
- The solution limits code reuse | ||
- Any change in a processing filter step will lead to a release of all the filters | ||
- The slowest processing filter step can become a bottleneck impacting the overall throughput of the service | ||
- As the solution scales, it will scale all the processing steps. Such scaling will lead to excessive resource utilisation in the cloud when not intended. | ||
|
||
Thus, the above approach is inflexible, non-scalable and against [reusability](https://en.wikipedia.org/wiki/Reusability){:target="_blank"}. A good solution must address all the above concerns. Let us try to find a solution this in next section. | ||
|
||
![Monolithic Service](https://raw.githubusercontent.com/Gaur4vGaur/traveller/master/images/patterns/2023-01-28-pipes-and-filters-pattern/patterns-pipes-and-filters-monolithic-service.png)*Monolithic Service with task modules* | ||
|
||
## Solution | ||
We can improve the suboptimal solution from the previous section by splitting the monolithic service into a series of components or functions, each performing a single processing filter step. These functions are combined to form a pipeline where functions receive a standard input and produce an accepted output. Such a decomposition will introduce loose coupling into the solution. It would allow effortless removal, replacement, rearrangement, or plug-in of a new filter step in pipelines. Moreover, it will enable code reuse if we need a similar processing filter in any other pipeline, the existing filters can be shared. | ||
|
||
![Pipes and Filters Solution](https://raw.githubusercontent.com/Gaur4vGaur/traveller/master/images/patterns/2023-01-28-pipes-and-filters-pattern/patterns-pipes-and-filters-solution.png)*Pipes and Filters Solution* | ||
|
||
The solution also addresses the issue of the slowest processing filter step being a bottleneck, as it provides an opportunity to scale individual functions. The slowest running function can run parallel instances to spread the load and improve throughput. The individual scaling of components will also save us from any extra costs. | ||
|
||
![Pipes and Filters Solution](https://raw.githubusercontent.com/Gaur4vGaur/traveller/master/images/patterns/2023-01-28-pipes-and-filters-pattern/patterns-pipes-and-filters-solution-scale.png)*Pipes and Filters Solution Scaling Individually* | ||
|
||
Finally, the processing filters can take advantage of cloud infrastructure. For instance, we can request a specific virtual machine based on the behaviour of a filter - processor-intensive or memory-intensive. | ||
|
||
## When to use this pattern | ||
As we now have a better knowledge of the pattern, this section explores when to think about the pattern. | ||
1. The pattern is valuable for breaking down complex processing workflows, where each step is independent. | ||
2. The processing steps have distinctive hardware needs or scalability requirements, and, in some cases, even preferred programming language requirements. | ||
3. The pattern can address flexibility concerns in workflows where requirements and business processes are changing constantly. | ||
4. The pattern is efficient in removing bottlenecks in workflows by parallel and distributed processing of individual steps. |
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.