Merge remote-tracking branch 'upstream/master'

Author: weisong44

docs/_blog/2017-08-27-announcing-the-release-of-apache-samza--0.13.1.md

@@ -0,0 +1,85 @@
+---
+layout: blog
+title: Announcing the release of Apache Samza 0.13.1
+icon: git-pull-request
+authors:
+    - name: Navina
+      website: 
+      image: 
+excerpt_separator: <!--more-->
+---
+<!--
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+-->
+
+
+Testing the excerpt
+
+<!--more-->
+
+
+Announcing the release of Apache Samza 0.13.1
+
+We are very excited to announce the release of **Apache Samza 0.13.1**  
+Samza has been powering real-time applications in production across several large companies (including LinkedIn, Netflix, Uber) for years now. Samza provides leading support for large-scale stateful stream processing with
+
+-   First class support for local state (with RocksDB store). This allows a stateful application to scale up to 1.1 Million events/sec on a single machine with SSD.
+-   Support for incremental checkpointing of state instead of full snapshots. This enables Samza to scale to applications with very large state.
+-   A fully pluggable model for input sources (e.g. Kafka, Kinesis, DynamoDB streams etc.) and output systems (HDFS, Kafka, ElastiCache etc.).
+-   A fully asynchronous programming model that makes parallelizing remote calls efficient and effortless.
+-   High level API for expressing complex stream processing pipelines in a few lines of code.
+-   Flexible deployment model for running the the applications in any hosting environment and with cluster managers other than YARN.
+-   Features like canaries, upgrades and rollbacks that support extremely large deployments with minimal downtime.
+
+
+### Enhancements, Upgrades and Bug Fixes 
+
+This is a stability release to make Samza as an embedded library production ready. Samza as a library is part of Samza’s Flexible Deployment model; release fixes a number of outstanding bugs includes the following enhancements to existing features:
+
+-   **Standalone**
+-   [SAMZA-1165](https://issues.apache.org/jira/browse/SAMZA-1165) Cleanup data created by ZkStandalone in ZK
+-   [SAMZA-1324](https://issues.apache.org/jira/browse/SAMZA-1324) Add a metrics reporter lifecycle for JobCoordinator component of StreamProcessor
+-   [SAMZA-1336](https://issues.apache.org/jira/browse/SAMZA-1336) Standalone session expiration propagation
+-   [SAMZA-1337](https://issues.apache.org/jira/browse/SAMZA-1337) LocalApplicationRunner supports StreamTask
+-   [SAMZA-1339](https://issues.apache.org/jira/browse/SAMZA-1339) Add standalone integration tests
+-   **General**
+-   [SAMZA-1282](https://issues.apache.org/jira/browse/SAMZA-1282) Fix killed leader process issue when spinning up more containers than the number of tasks kills leader
+-   [SAMZA-1340](https://issues.apache.org/jira/browse/SAMZA-1340) StreamProcessor does not propagate container failures from StreamTask
+-   [SAMZA-1346](https://issues.apache.org/jira/browse/SAMZA-1346) GroupByContainerCount.balance() should guard against null LocalityManager
+-   [SAMZA-1347](https://issues.apache.org/jira/browse/SAMZA-1347) GroupByContainerIds NPE if containerIds list is null
+-   [SAMZA-1358](https://issues.apache.org/jira/browse/SAMZA-1358) task.class empty string should be ignored when app.class is configured
+-   [SAMZA-1361](https://issues.apache.org/jira/browse/SAMZA-1361) OperatorImplGraph used wrong keys to store/retrieve OperatorImpl in the map
+-   [SAMZA-1366](https://issues.apache.org/jira/browse/SAMZA-1366) ScriptRunner should allow callers to control the child process environment
+-   [SAMZA-1384](https://issues.apache.org/jira/browse/SAMZA-1384) Race condition with async commit affects checkpoint correctness
+-   [SAMZA-1385](https://issues.apache.org/jira/browse/SAMZA-1385) Fix coordination issues during stream creation in LocalApplicationRunner
+
+Overall, [29 JIRAs](https://issues.apache.org/jira/issues/?jql=project%20%3D%2012314526%20AND%20fixVersion%20%3D%2012340845%20ORDER%20BY%20priority%20DESC%2C%20key%20ASC) were resolved in this release. 
+A source download of the 0.13.1 release is available [here](http://www.apache.org/dyn/closer.cgi/samza/0.13.1). The release JARs are also available in Apache’s Maven repository. See Samza’s [download](http://samza.apache.org/startup/download/) page for details and Samza’s [feature preview](https://samza.apache.org/startup/preview/) for new features. We requires JDK version newer than 1.8.0_111 when running 0.13.1 release for users who are using Scala 2.12.
+
+### Community Developments
+
+We’ve made great community progress since the last release (0.13.0). We presented Samza high level API features at the Cloud+Data NEXT Conference 2017 held in Silicon Valley, USA, and also gave a talk regarding the key features (Secret Kung Fu) of Samza at ArchSummit 2017 in Shenzhen, China, and a detailed study of stateful stream processing in VLDB 2017. Here are the details to these conferences.
+
+-   July 15, 2017 - [Unified Processing with the Samza High-level API (Cloud+Data NEXT Conference, Silicon Valley)](http://www.cdnextcon.com/recap.html) [slides](https://www.slideshare.net/YiPan7/nextcon-samza-preso-july-final)
+-   July 7, 2017 - [Secret Kung Fu of Massive Scale Stream Processing with Apache Samza - Xinyu Liu](http://sz2017.archsummit.com/presentation/900) [ArchSummit, Shenzhen, 2017]
+-   Aug 28, 2017 - [Samza: Stateful Scalable Stream Processing at LinkedIn - Kartik Paramasivam (ACM VLDB, Munich, 2017)](http://www.vldb.org/pvldb/vol10/p1634-noghabi.pdf)
+
+In industry, Samza got new adopters, including Redfin and VMWare. 
+As future development, we are continuing working on improving the new High Level API and flexible deployment features. Here is the list of the tasks for upcoming features and improvements.
+### Contribute
+
+It’s a great time to get involved. You can start by reviewing the [tutorials](http://samza.apache.org/startup/preview/#try-it-out), signing up for the [mailing list](http://samza.apache.org/community/mailing-lists.html), and grabbing some [newbie JIRAs](https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20labels%20%3D%20newbie%20AND%20status%20%3D%20Open).  
+I’d like to close by thanking everyone who’s been involved in the project. It’s been a great experience to be involved in this community, and I look forward to its continued growth.

docs/_blog/2018-07-19-stream_processing-meetup-summary.md

@@ -0,0 +1,70 @@
+---
+layout: blog
+title: Recap of Stream Processing with Apache Kafka & Apache Samza (July '18)
+icon: analytics
+authors:
+    - name:
+      website: 
+      image: 
+excerpt_separator: <!--more-->
+---
+<!--
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+-->
+
+A look back at July edition of quarterly stream processing meetup
+
+<!--more-->
+
+
+On July 19th another successful stream processing meetup was hosted by LinkedIn! 
+This event focused on Apache Kafka, Apache Samza, and related streaming technologies. 
+This meetup was a full house and had techincal deep dives by engineers from LinkedIn and Uber on the latest
+and greatest in streaming tech
+
+<br>
+
+
+### [Beam me up Samza: How we built a Samza Runner for Apache Beam](https://youtu.be/o5GaifLoZho)
+
+LinkedIn's Xinyu Liu presented [Beam me up Samza](https://bit.ly/2Nyc4pl), describing how Linkedin is harnessing cutting edge features of Beam. 
+Apache Beam provides an easy-to-use, and powerful model for state-of-the-art stream and batch processing, portability 
+across a variety of languages, and the ability to converge offline and nearline data processing. In this talk,
+he discussed the Beam API and its implementation in Samza and the benefits of Beam Runner to the Samza and Beam community.
+He also explored various use cases of Beam at LinkedIN and future work on it. 
+
+
+### [uReplicator: Uber Engineering’s Scalable Robust Kafka Replicator](https://bit.ly/2NxvFpz)
+
+
+Uber operates more than 20 Kafka clusters to collect system, application logs and event data from rider and driver apps. 
+Uber's Hongliang Xu shared his insignts on Uber's approch for replicating data between Kafka clusters across multiple data centers. 
+He covered the history behind [uReplicator](https://bit.ly/2NxvFpz) and gave the high level architecture. Furthermore he also discussed the
+scalability challenges and operational overhead as the Uber exapanded and how did they build Federated uReplicator 
+which addressed challanges at scale
+
+
+### [Concourse - Near real time notifications platform at Linkedin](https://youtu.be/Fszo6jThq0I)
+
+
+[Concourse](https://bit.ly/2zXNwUJ) is LinkedIn’s first near-real-time targeting and scoring platform for notifications. In this talk LinkedIn's Ajith Muralidharan & Vivek Nelamangala provided an in-depth overview of the design and various scaling optimizations. 
+Concourse has an ability to score millions of notifications per second, while supporting the use of feature-rich machine learning 
+models based on terabytes of feature data.
+
+
+<!--more-->
+
+

docs/_blog/2018-09-26-stangeloop-recap.md

@@ -0,0 +1,50 @@
+---
+layout: blog
+title: Recap of Chasing Stream Processing Utopia
+icon: analytics
+authors:
+    - name:
+      website: 
+      image: 
+excerpt_separator: <!--more-->
+---
+<!--
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+-->
+
+Strange Loop, St. Louis, MO
+
+<!--more-->
+
+Over the last 15 years batch processing frameworks have thrived and ruled over big data processing. But now in the age of social computing, it is no longer acceptable to wait for data to land into a data-lake before it gets processed. 
+We want our applications to react to new data as soon as it gets generated upstream. For a web site, members expect their feed to be updated as soon as some relevant activity, news, jobs etc. happens. 
+We are talking seconds (or minutes). We also want to detect degraded site experience, fraud, security breaches, spam etc. instantaneously. Even business metrics (written in traditionally batch oriented languages like HIVE/PIG) are now expected to run in realtime. The current status-quo of real-time data processing (stream processing) is still very far from Utopia.
+
+Kartik Paramasivam, The Director of Engineering presented Chasing the Stream Utopia at Strange Loop '18. The talk was inspired 
+by the extensive growth in Streaming Data at Linkedin, which has experienced a growth of as high as 5 Trillion Messages per day in 2018. 
+Linkedin supports close to 3000 applications in production using Kafka and Samza. He shed further light on Samza's claim
+as State of the art Stream Processing framework in the streaming world, supporting use cases at LinkedIn, Slack, Uber, Intuit etc
+
+His talk described LinkedIn's path on Chasing Utopia in Streaming world running apps at any complexity, any scale, 
+any source, any language, and any environment! He shed light on all of the above with actual use cases from LinkedIn using Samza and Kafka in production. He touched Samza's battle tested Stateful and Stateless processing, and also on the 
+newer available features like event time based processing using Beam Runner for Samza and Samza SQL. He further briefly explained running 
+and managing Kafka at Scale. Covering an array of topics from Kafka Cluster Management Woes to Dynamic Load Balancing 
+using Kafka Cruise Control. 
+
+He further added the tooling ecosystem that supports these apps and streaming challanges that are faced at LinkedIn. He
+concluded with the upcoming releases and features of Samza (Apache Samza 1.0) and Kafka (Apache Kafka 2.0). Please find more [here] (https://youtu.be/2y8QImf-RpI)
+
+<br>

docs/_case-studies/digitalsmiths.md

@@ -5,6 +5,7 @@ title: Totally awesome use-case of samza by DigitalSmiths # title of case study
 study_domain: digitalsmiths.com # just the domain, not the protocol
 priority: 7
 menu_title: DigitalSmiths # what shows up in the menu
+exclude_from_loop: true
 excerpt_separator: <!--more-->
 ---
 <!--

docs/_case-studies/ebay.md

@@ -1,7 +1,7 @@
 ---
 layout: case-study
 hide_title: true # so we have control in case-study layout, but can still use page
-title: Low Latency Web Scale Fraud Prevention
+title: Low Latency Web-Scale Fraud Prevention
 study_domain: ebay.com
 menu_title: eBay
 excerpt_separator: <!--more-->
@@ -23,34 +23,47 @@ excerpt_separator: <!--more-->
    limitations under the License.
 -->
 
-Low Latency Web Scale Fraud Prevention
+How Samza powers low-latency, web-scale fraud prevention at Ebay?
 
 <!--more-->
 
-eBay Enterprise is the world’s largest omni-channel commerce provider with 
-hundreds millions of units shipped annually, as commerce gets more 
-convenient and complex, so does fraud. The engineering team at eBay 
-Enterprise selected Samza as the platform to build the horizontally 
-scalable, realtime (sub-seconds) and fault tolerant abnormality detection 
-system. For example, the system computes and evaluates key metrics to 
-detect abnormal behaviors
-
--   Transaction velocity (#tnx/day) and change (#tnx/day vs #tnx/day over n days)
--   Amount velocity ($tnx/day) and change ($tnx/day vs $tnx/day over n days)
-
-A wide range of realtime and historical adjunct data from various sources 
-including people, places, interests, social and connections are ingested 
-through Kafka, and stored in local RocksDB state store with changelog 
-enabled for recovery. Incoming transaction data is aggregated using 
-windowing and then joined with adjunct data stores in multiple stages. 
-The system generates potential fraud cases for review real time. Finally, 
-the engineering team at eBay Enterprise has built an OpenTSDB and Grafana 
-based monitoring system using metrics collected through JMX.
-
-Key Samza features: *Stateful processing*, *Windowing*, *Kafka-integration*,
-*JMX-metrics*
-
-More information
-
--   [https://www.slideshare.net/edibice/extremely-low-latency-web-scale-fraud-prevention-with-apache-samza-kafka-and-friends](https://www.slideshare.net/edibice/extremely-low-latency-web-scale-fraud-prevention-with-apache-samza-kafka-and-friends)
--   [http://ebayenterprise.com/](http://ebayenterprise.com/)
+eBay Enterprise is the world’s largest omni-channel commerce provider. The engineering team at eBay chose Apache Samza to build _PreCog_, their 
+horizontally scalable anomaly detection system. 
+
+_PreCog_ extensively leverages Samza's high-performance, fault-tolerant local storage. Its architecture had the following requirements, for which Samza perfectly fit the bill: <br/>
+
+_Web-scale:_ Scale to a large number of users and large volume of data per-user. Additionally, should be possible to add more commodity hardware and scale horizontally. <br/>
+_Low-latency:_ Process customer interactions real-time by reacting in milliseconds instead of hours. <br/>
+_Fault-tolerance:_ Gracefully tolerate and handle hardware failures. <br/>
+
+![diagram-large](/img/{{site.version}}/learn/documentation/case-study/ebay.png)
+
+The PreCog anomaly-detection system comprises of multiple tiers, with each tier consisting of multiple Samza jobs, which process the output of the previous tier.
+
+_Ingestion tier:_ In this tier, a variety of historical and realtime data from various
+sources including people, places etc., is ingested into Kafka.
+
+_Fanout tier:_ This tier consists of Samza jobs which process the Kafka events, fan them out and re-partition them based on various
+facets like email-address, ip-address, credit-card number, shipping address etc. 
+
+_Compute tier:_ The Samza jobs in this tier consume messages from the fan-out tier and compute various key metrics and derived features. Features used to evaluate fraud include: 
+
+1. Number of transactions per-customer per-day <br/>
+2. Change in the number of daily transactions over the past few days <br/>
+3. Amount value ($$) of each transaction per-day <br/>
+4. Change in the amount value of transactions over a sliding time-window <br/>
+5. Number of transactions per shipping-address
+
+_Assembly tier:_ This tier comprises of Samza jobs which join the output of the compute-tier with other additional data-sources
+and make a final determination on transaction-fraud. 
+
+For monitoring the _PreCog_ pipeline, EBay leverages Samza's [JMXMetricsReporter](/learn/documentation/{{site.version}}/operations/monitoring.html) and ingests the reported metrics into OpenTSDB/ HBase. The metrics are then 
+visualzed using [Grafana](https://grafana.com/).
+
+
+Key Samza features: *Stateful processing*, *Windowing*, *Kafka-integration*, *JMX-metrics*
+
+More information:
+
+-   [Slides: Low latency Fraud prevention with Apache Samza](https://www.slideshare.net/edibice/extremely-low-latency-web-scale-fraud-prevention-with-apache-samza-kafka-and-friends)
+-   [http://ebayenterprise.com/](http://ebayenterprise.com/)

docs/_case-studies/fortscale.md

@@ -5,6 +5,7 @@ title: Totally awesome use-case of samza by FortScale # title of case study page
 study_domain: fortscale.com # just the domain, not the protocol
 priority: 6
 menu_title: FortScale # what shows up in the menu
+exclude_from_loop: true
 excerpt_separator: <!--more-->
 ---
 <!--

docs/_case-studies/index.md

@@ -20,7 +20,7 @@ exclude_from_loop: true
    limitations under the License.
 -->
 
-Explore the many use-cases of the Samza Framework via our case-studies.
+Explore the many use-cases of the Samza via our case-studies. For a complete list of companies using Samza, visit our <a href="/powered-by/">Powered By</a> page
 
 <ul class="case-studies">
 

docs/_case-studies/intuit.md

@@ -5,6 +5,7 @@ title: Totally awesome use-case of samza by Intuit # title of case study page
 study_domain: intuit.com # just the domain, not the protocol
 priority: 5
 menu_title: Intuit # what shows up in the menu
+exclude_from_loop: true
 excerpt_separator: <!--more-->
 ---
 <!--