-
Notifications
You must be signed in to change notification settings - Fork 0
/
a-hackish-system-command-service-in-go.html
42 lines (42 loc) · 10.9 KB
/
a-hackish-system-command-service-in-go.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
<!doctype html><html lang=en><head><meta charset=utf-8><meta name=mobile-web-app-capable content="yes"><meta name=viewport content="width=device-width,initial-scale=1"><title>A Hackish System Command Service in Go - Adam Drake
</title><meta name=description content="Adam Drake is an advisor to scale-up tech companies. He writes about ML/AI/crypto/data, leadership, and building tech teams."><link rel="shortcut icon" href=https://adamdrake.com/static/favicon.ico><link rel=authorization_endpoint href=https://indieauth.com/auth><link rel=token_endpoint href=https://tokens.indieauth.com/token><link rel=me href=https://github.com/adamdrake><link rel=stylesheet href=https://adamdrake.com/css/style.min.css crossorigin=anonymous media=screen><meta property="og:url" content="https://adamdrake.com/"><meta property="og:title" content="Adam Drake"><meta property="og:site_name" content="Adam Drake"><meta property="og:type" content="website"><meta property="og:description" content="Adam Drake is an advisor to scale-up tech companies. He writes about ML/AI/crypto/data, leadership, and building tech teams."><meta property="og:image" content="/static/images/twitter-card.jpg"><meta name=twitter:title content="Adam Drake"><meta name=twitter:description content="Adam Drake is an advisor to scale-up tech companies. He writes about ML/AI/crypto/data, leadership, and building tech teams."><meta name=twitter:card content="summary_large_image"><meta name=twitter:image content="/static/images/twitter-card.jpg"></head><body><header><section><div class="header flex row"><div class="header__item flex row"><a id=site__name href=https://adamdrake.com/>Adam Drake</a></div><div class="flex row"><nav aria-label="page menu" class="flex row"><ul role=menubar class="flex row"><li role=none><a class=sidebar-nav-itemmenu__item href=/ title>Latest</a></li><li role=none><a class=sidebar-nav-itemmenu__item href=/about.html title>About</a></li><li role=none><a class=sidebar-nav-itemmenu__item href=/cases.html title>Case Studies</a></li><li role=none><a class=sidebar-nav-itemmenu__item href=/contact.html title>Contact</a></li><li role=none><a class="sidebar-nav-item activemenu__item" href=/posts.html title=Posts>Posts</a></li><li role=none><a class=sidebar-nav-itemmenu__item href=/press.html title>Press</a></li><li><button class="subscribe subscribe-btn">
<a href=https://www.digitalmaneuver.com/#/portal>Subscribe to newsletter</a></button></li></ul></nav></div></div></section></header><main aria-role=main><section><ul id=feed__ul><li class="feed__li h-entry"><div class=feed__content><time class="hidden dt-published">2015-09-27 00:00:00 +0000 UTC</time><div class="flex properties__row"><div rel=author class="flex left p-author h-card hidden"><img class=u-photo src=https://adamdrake.com/static/images/adam_drake_240.jpg alt="Adam Drake" id=author-img><div><p rel=me class=p-name id=author-name>Adam Drake</p><p class=properties>Sep 27, 2015</p></div></div><div class="flex right properties"></div></div><article class="md p-summary e-content"><h1 class=p-name>A Hackish System Command Service in Go</h1><p>When building data products, often the end result is either a recurring report or an API which allows for an interface with other services. In these cases, the product might be run by a cron job, or even manually, depending on the situation. When the product is part of (more frequently, at the end of) a larger data processing pipeline, this can become inefficient or problematic since steps earlier in the pipeline can cause the product to supply incorrect information or fail altogether.</p><p>To get around this problem it is often desirable to have previous steps in the data processing pipeline directly trigger the data product to run. This ensures that earlier steps of the pipeline have completed successfully and also eliminates the need for buffer time before the cron job starts. A way this coupling is often accomplished is via an HTTP endpoint which can be hit by some calling entity, thus triggering the processing job to start.</p><p>When the cron job is already running, building such an HTTP endpoint can be very straightforward since you need only run a local command on the system in question (responding with HTTP code 200, OK) and then while the command is running any subsequent requests can provide an HTTP 503 (Service Unavailable) response.</p><p>With that in mind, here is a small example in Go which accomplishes the task. It will listen on some port for requests to a <code>/start</code> endpoint. When that URL is accessed the command will be run in a separate goroutine, and any requests which come in while the command is running will receive a 503 response. After the command completes and errors are handled the system is available and can be restarted again.</p><p>This is just a very rough outline and has some nasty things (e.g., global variables, security problems) but it provides the necessary endpoint and does what is needed with minimal effort.</p><div class=highlight><pre tabindex=0 style=color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-Go data-lang=Go><span style=display:flex><span><span style=color:#f92672>package</span> <span style=color:#a6e22e>main</span>
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:#f92672>import</span> (
</span></span><span style=display:flex><span> <span style=color:#e6db74>"net/http"</span>
</span></span><span style=display:flex><span> <span style=color:#e6db74>"os/exec"</span>
</span></span><span style=display:flex><span>)
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:#66d9ef>const</span> (
</span></span><span style=display:flex><span> <span style=color:#a6e22e>command</span> = <span style=color:#e6db74>"commandTorun"</span>
</span></span><span style=display:flex><span> <span style=color:#a6e22e>args</span> = <span style=color:#e6db74>"argumentsForTheCommand"</span>
</span></span><span style=display:flex><span> <span style=color:#a6e22e>port</span> = <span style=color:#e6db74>":8080"</span>
</span></span><span style=display:flex><span>)
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:#66d9ef>var</span> <span style=color:#a6e22e>jobRunning</span> = <span style=color:#66d9ef>false</span>
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:#66d9ef>func</span> <span style=color:#a6e22e>runJob</span>() {
</span></span><span style=display:flex><span> <span style=color:#a6e22e>jobRunning</span> = <span style=color:#66d9ef>true</span>
</span></span><span style=display:flex><span> <span style=color:#a6e22e>err</span> <span style=color:#f92672>:=</span> <span style=color:#a6e22e>exec</span>.<span style=color:#a6e22e>Command</span>(<span style=color:#a6e22e>command</span>, <span style=color:#a6e22e>args</span>).<span style=color:#a6e22e>Run</span>()
</span></span><span style=display:flex><span> <span style=color:#66d9ef>if</span> <span style=color:#a6e22e>err</span> <span style=color:#f92672>!=</span> <span style=color:#66d9ef>nil</span> {
</span></span><span style=display:flex><span> <span style=color:#75715e>// Error was returned from system command. Do something.
</span></span></span><span style=display:flex><span><span style=color:#75715e></span> }
</span></span><span style=display:flex><span> <span style=color:#a6e22e>jobRunning</span> = <span style=color:#66d9ef>false</span>
</span></span><span style=display:flex><span>}
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:#66d9ef>func</span> <span style=color:#a6e22e>jobStartEndpoint</span>(<span style=color:#a6e22e>w</span> <span style=color:#a6e22e>http</span>.<span style=color:#a6e22e>ResponseWriter</span>, <span style=color:#a6e22e>r</span> <span style=color:#f92672>*</span><span style=color:#a6e22e>http</span>.<span style=color:#a6e22e>Request</span>) {
</span></span><span style=display:flex><span> <span style=color:#66d9ef>if</span> <span style=color:#a6e22e>jobRunning</span> {
</span></span><span style=display:flex><span> <span style=color:#a6e22e>w</span>.<span style=color:#a6e22e>WriteHeader</span>(<span style=color:#ae81ff>503</span>)
</span></span><span style=display:flex><span> } <span style=color:#66d9ef>else</span> {
</span></span><span style=display:flex><span> <span style=color:#66d9ef>go</span> <span style=color:#a6e22e>runJob</span>()
</span></span><span style=display:flex><span> <span style=color:#a6e22e>w</span>.<span style=color:#a6e22e>WriteHeader</span>(<span style=color:#ae81ff>200</span>)
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span> }
</span></span><span style=display:flex><span>}
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span><span style=color:#66d9ef>func</span> <span style=color:#a6e22e>main</span>() {
</span></span><span style=display:flex><span> <span style=color:#a6e22e>http</span>.<span style=color:#a6e22e>HandleFunc</span>(<span style=color:#e6db74>"/start"</span>, <span style=color:#a6e22e>jobStartEndpoint</span>)
</span></span><span style=display:flex><span> <span style=color:#a6e22e>http</span>.<span style=color:#a6e22e>ListenAndServe</span>(<span style=color:#a6e22e>port</span>, <span style=color:#66d9ef>nil</span>)
</span></span><span style=display:flex><span>}
</span></span></code></pre></div><a class=hidden href=https://brid.gy/publish/mastodon></a><a class=hidden href=https://brid.gy/publish/twitter></a><a class=hidden href=https://fed.brid.gy/></a><data class=p-bridgy-omit-link value=false></data></article></div><div id=webmentions></div></li></ul></section></main><hr><footer class="flex col"><section class="footer-bio content"><p><strong>Adam Drake</strong> leads technical business transformations in global and multi-cultural environments. He has a passion for helping companies become more productive by improving internal leadership capabilities, and accelerating product development through technology and data architecture guidance. Adam has served as a White House Presidential Innovation Fellow and is an IEEE Senior Member.</p></section><button class="subscribe subscribe-btn">
<a href=https://www.digitalmaneuver.com/#/portal>Subscribe to newsletter</a></button><div class=social-icons><a rel=me href=https://github.com/adamdrake title=GitHub><svg width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentcolor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-github"><path d="M9 19c-5 1.5-5-2.5-7-3m14 6v-3.87a3.37 3.37.0 00-.94-2.61c3.14-.35 6.44-1.54 6.44-7A5.44 5.44.0 0020 4.77 5.07 5.07.0 0019.91 1S18.73.65 16 2.48a13.38 13.38.0 00-7 0C6.27.65 5.09 1 5.09 1A5.07 5.07.0 005 4.77 5.44 5.44.0 003.5 8.55c0 5.42 3.3 6.61 6.44 7A3.37 3.37.0 009 18.13V22"/></svg></a></div></footer></body></html>