Skip to content

Commit 34a3985

Browse files
committed
pachyderm tool
1 parent 5f1e9f3 commit 34a3985

File tree

1 file changed

+20
-0
lines changed

1 file changed

+20
-0
lines changed

index.html

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -272,6 +272,26 @@ <h2 id="project_tagline">This page is a summary to keep the track of Hadoop rela
272272
<br> <a href="http://www.tinkerpop.com/">2. TinkerPop site</a></td>
273273
</tr>
274274

275+
<tr>
276+
<td width="20%">Pachyderm MapReduce</td>
277+
<td>Pachyderm is a completely new MapReduce engine built on top Docker and CoreOS.
278+
In Pachyderm MapReduce (PMR) a job is an HTTP server inside a Docker container
279+
(a microservice). You give Pachyderm a Docker image and it will automatically
280+
distribute it throughout the cluster next to your data. Data is POSTed to
281+
the container over HTTP and the results are stored back in the file system.
282+
You can implement the web server in any language you want and pull in any library.
283+
Pachyderm also creates a DAG for all the jobs in the system and their dependencies
284+
and it automatically schedules the pipeline such that each job isn’t run until it’s
285+
dependencies have completed. Everything in Pachyderm “speaks in diffs” so it knows
286+
exactly which data has changed and which subsets of the pipeline need to be rerun.
287+
CoreOS is an open source lightweight operating system based on Chrome OS, actually
288+
CoreOS is a fork of Chrome OS. CoreOS provides only the minimal functionality
289+
required for deploying applications inside software containers, together with
290+
built-in mechanisms for service discovery and configuration sharing</td>
291+
<td width="20%"><a href="http://www.pachyderm.io/">1. Pachyderm site</a>
292+
<br> <a href="https://medium.com/pachyderm-data/lets-build-a-modern-hadoop-4fc160f8d74f">2. Pachyderm introduction article</a></td>
293+
</tr>
294+
275295

276296
<!---------------------------->
277297
<!-- NoSQL ecosystem -->

0 commit comments

Comments
 (0)