{
  "trx_id": "90f991a2aef0117e9b3986521415283a4899b5cb",
  "block": 40265902,
  "trx_in_block": 18,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2020-01-26T12:29:45",
  "op": [
    "comment",
    {
      "parent_author": "petecorey",
      "parent_permlink": "test",
      "author": "steemitboard",
      "permlink": "steemitboard-notify-petecorey-20200126t122945000z",
      "title": "",
      "body": "Congratulations @petecorey! You received a personal award!\n\n<table><tr><td>https://steemitimages.com/70x70/http://steemitboard.com/@petecorey/birthday3.png</td><td>Happy Birthday! - You are on the Steem blockchain for 3 years!</td></tr></table>\n\n<sub>_You can view [your badges on your Steem Board](https://steemitboard.com/@petecorey) and compare to others on the [Steem Ranking](https://steemitboard.com/ranking/index.php?name=petecorey)_</sub>\n\n\n###### [Vote for @Steemitboard as a witness](https://v2.steemconnect.com/sign/account-witness-vote?witness=steemitboard&approve=1) to get one more award and increased upvotes!",
      "json_metadata": "{\"image\":[\"https://steemitboard.com/img/notify.png\"]}"
    }
  ]
}

{
  "trx_id": "2e6783506f6dca65ffe018bc9ec74520071a85f9",
  "block": 29795055,
  "trx_in_block": 4,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2019-01-26T13:50:18",
  "op": [
    "comment",
    {
      "parent_author": "petecorey",
      "parent_permlink": "test",
      "author": "steemitboard",
      "permlink": "steemitboard-notify-petecorey-20190126t135017000z",
      "title": "",
      "body": "Congratulations @petecorey! You received a personal award!\n\n<table><tr><td>https://steemitimages.com/70x70/http://steemitboard.com/@petecorey/birthday2.png</td><td>Happy Birthday! - You are on the Steem blockchain for 2 years!</td></tr></table>\n\n<sub>_[Click here to view your Board](https://steemitboard.com/@petecorey)_</sub>\n\n\n> Support [SteemitBoard's project](https://steemit.com/@steemitboard)! **[Vote for its witness](https://v2.steemconnect.com/sign/account-witness-vote?witness=steemitboard&approve=1)** and **get one more award**!",
      "json_metadata": "{\"image\":[\"https://steemitboard.com/img/notify.png\"]}"
    }
  ]
}

{
  "trx_id": "b4cce96303af6f46af856f8a8b08b4e33e449fb8",
  "block": 23762942,
  "trx_in_block": 15,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2018-06-30T02:16:03",
  "op": [
    "comment",
    {
      "parent_author": "",
      "parent_permlink": "test",
      "author": "petecorey",
      "permlink": "test",
      "title": "this is a test",
      "body": "lk",
      "json_metadata": "{\"tags\":[\"test\"],\"app\":\"steemit/0.1\",\"format\":\"markdown\"}"
    }
  ]
}

{
  "trx_id": "d6ae321a487f088f86860a96969649535b39b039",
  "block": 23761440,
  "trx_in_block": 47,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2018-06-30T01:00:57",
  "op": [
    "vote",
    {
      "voter": "fastresteem",
      "author": "petecorey",
      "permlink": "test",
      "weight": 100
    }
  ]
}

{
  "trx_id": "b5035f9198e717f3afd693ca4641966e48747a45",
  "block": 23761436,
  "trx_in_block": 25,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2018-06-30T01:00:45",
  "op": [
    "comment",
    {
      "parent_author": "",
      "parent_permlink": "test",
      "author": "petecorey",
      "permlink": "test",
      "title": "this is a test",
      "body": "---\nlayout: post\ntitle:  \"Making Noise with J\"\ndescription: \"Let's try to make music with the J programming language and a handful of other helpful tools and utilities.\"\nauthor: \"Pete Corey\"\ndate:   2018-07-02\ntags: [\"J\", \"Music\"]\nrelated: []\n---\n\nI've always been fascinated by live-coded music. Frameworks like [Chuck](http://chuck.cs.princeton.edu/), [Supercollider](https://supercollider.github.io/), [Overtone](http://overtone.github.io/), [Extempore](https://extemporelang.github.io/), and [Sonic PI](https://sonic-pi.net/), along with popular performers and musicians like [Sam Aaron](https://www.youtube.com/watch?v=KJPdbp1An2s) and [Andrew Sorensen](https://www.youtube.com/watch?v=GSGKEy8vHqg) have never ceased to amaze and inspire me.\n\nThat said, whenever I've tried to use one of those tools or frameworks to create my own music, I've always quickly given up. Maybe it's because I'm just lazy and learning new things is hard, but I've always told myself that it's because the tools I was using _just didn't fit_ with how I felt programming music should be. Syntactically, ergonomically, and conceptually, the tools just didn't jive.\n\nAnd then I stumbled across [J](http://jsoftware.com/).\n\nJ and [the entire family of APL languages](https://en.wikipedia.org/wiki/APL_(programming_language)) have a beautiful terseness and closeness to the data being operated on. They're also fundamentally designed to operate on arrays, a data structure ripe for musical interpretation. I've convinced myself that if I can learn J, I'll be able to build the live coding environment of my dreams!\n\nThat's a big goal, but I'm taking baby steps to get there. Today, I'll show you how I managed to make noise with J.\n\n## Making Noise Without J\n\nMy plan for making noise with J doesn't actually involve my J software producing any noise directly. Instead, it'll act as a controller that instructs other software on my machine to make noise on its behalf.\n\nThe software making the noise will be [SimpleSynth](http://notahat.com/simplesynth/), which is a small, easy to use MIDI synthesizer. If you're following along, feel free to use any other MIDI synth you'd like, or a full audio workstation like [Ableton](https://www.ableton.com/en/) or even [GarageBand](https://www.apple.com/mac/garageband/).\n\n<div style=\"width: 60%; margin: 2em auto;\">\n  <img src=\"https://s3-us-west-1.amazonaws.com/www.east5th.co/img/making-noise-with-j/1.png\" style=\" width: 100%;\"/>\n  <p style=\"text-align: center; color: #ccc; margin: 0;\">SimpleSynth.</p>\n</div>\n\nWhen we fire up SimpleSynth, it'll ask which MIDI source it should use. [MIDI](https://en.wikipedia.org/wiki/MIDI) is a protocol that lets us pass around musical information, like when and how loud certain notes should be played, between different devices. SimpleSynth is asking which stream of notes it should listen to and play.\n\n<div style=\"width: 60%; margin: 2em auto;\">\n  <img src=\"https://s3-us-west-1.amazonaws.com/www.east5th.co/img/making-noise-with-j/3.png\" style=\" width: 100%;\"/>\n  <p style=\"text-align: center; color: #ccc; margin: 0;\">Setting up our J virtual device in MIDI Studio.</p>\n</div>\n\nI used MacOS' built-in MIDI Studio to create a virtual MIDI channel called \"J\", with a MIDI port called \"Bus 1.\" After making sure the virtual device was online, I selected it in SimpleSynth.\n\n<div style=\"width: 60%; margin: 2em auto;\">\n  <img src=\"https://s3-us-west-1.amazonaws.com/www.east5th.co/img/making-noise-with-j/2.png\" style=\" width: 100%;\"/>\n  <p style=\"text-align: center; color: #ccc; margin: 0;\">Selecting our J virtual device in SimpleSynth.</p>\n</div>\n\nThe last piece of the puzzle is finding some way of programmatically sending MIDI messages through my \"J Bus 1\" to be played by SimpleSynth. Geert Bevin's [SendMIDI](https://github.com/gbevin/SendMIDI) command line tool did just the trick. \n\nOnce installed, we can use SendMIDI to send MIDI notes to SimpleSynth from our command line:\n\n<pre class='language-j'><code class='language-j'>sendmidi dev \"J Bus 1\" on 60 100\n</code></pre>\n\nTurning `on`{:.language-j} note `60`{:.language-j}, with a velocity of `100`{:.language-j} effectively plays a middle C at full volume.\n\nNow we're making music!\n\n## Talking to SendMIDI with J\n\nThe next challenge lies in getting J to execute `sendmidi`{:.language-j} commands.\n\nAfter much searching and head scratching, I learned that J exposes [a wide range of miscellaneous functionality](https://twitter.com/seanstickle/status/1000766939193634825) under [the \"foreigns\" (`!:`{:.language-j}) verb](http://www.jsoftware.com/help/dictionary/d412.htm). Calling `2!:1 y`{:.language-j} lets you spawn a new process, running whatever command you pass in through `y`{:.language-j}.\n\nLet's try invoking our spawn verb with our `sendmidi`{:.language-j} command:\n\n<pre class='language-j'><code class='language-j'>   2!:1 'sendmidi dev \"J Bus 1\" on 60 100'\n|interface error\n|       2!:1'sendmidi dev \"J Bus 1\" on 60 100'\n</code></pre>\n\nAfter even more searching and head scratching, I realized that I needed to use the fully-qualified `sendmidi`{:.language-j} path when making the call:\n\n<pre class='language-j'><code class='language-j'>   2!:1 '/usr/local/bin/sendmidi dev \"J Bus 1\" on 60 100'\n</code></pre>\n\nI hear sound! Success!\n\n## Making Music with J\n\nWhile this is great, it's not much better just running our `sendmidi`{:.language-j} command directly from the command line. What would make things even better is if we could build ourselves a `play`{:.language-j} verb that plays any notes passed to it.\n\nFor example, if I were to run:\n\n<pre class='language-j'><code class='language-j'>   play 60 64 67\n</code></pre>\n\nI'd expect J to construct and execute our `sendmidi`{:.language-j} command, which should play a C major chord:\n\n<pre class='language-j'><code class='language-j'>sendmidi dev \"J Bus 1\" on 60 100 on 64 100 on 67 100\n</code></pre>\n\nAfter a few brain-expanding weekends of playing around in J, I came up with this version of the `play`{:.language-j} verb:\n\n<pre class='language-j'><code class='language-j'>   on =: ('on ',5|.' 100 ',\":)\"0\n   play =: [:2!:1'/usr/local/bin/sendmidi dev \"J Bus 1\" ',[:,/on\n</code></pre>\n\nThe `on`{:.language-j} verb turns an integer note into an \"on string\" of the format, `'on <note> 100 '`{:.language-j}, and the `play`{:.language-j} verb spawns the result of appending `'/usr/local/bin/sendmidi ...'`{:.language-j} to append mapped over `on`{:.language-j} applied to `y`{:.language-j}.\n\nPut simply, it constructs our `sendmidi`{:.language-j} command and executes it.\n\nWe can play a C major chord:\n\n<pre class='language-j'><code class='language-j'>   play 60 64 67\n</code></pre>\n\nOr any other chord we want:\n\n<pre class='language-j'><code class='language-j'>   play 60 63 54 70 73\n</code></pre>\n\n## Final Thoughts\n\nPlease keep in mind that I'm very new to J, and even newer to tacit programming. If you see anything that can be improved, clarified, or corrected, please [let me know](https://twitter.com/petecorey).\n\nI still feel very clunky and slow when it comes to using J. Building this two line program took hours of my time. That said, I feel like there is potential here. As I grow more used to the tacit paradigm and play with other ways of interacting to DAWs and other audio producers, I feel like J might turn into my ideal music creation environment.\n\nTime will tell.\n",
      "json_metadata": ""
    }
  ]
}

{
  "trx_id": "40aee545882eb747ba11a7aff96b35cfd1bcdb2e",
  "block": 23761406,
  "trx_in_block": 12,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2018-06-30T00:59:15",
  "op": [
    "vote",
    {
      "voter": "ax3",
      "author": "petecorey",
      "permlink": "test",
      "weight": 100
    }
  ]
}

{
  "trx_id": "cfc8d5d2e4ced0bc638a5ee94bc89595487a504f",
  "block": 23761402,
  "trx_in_block": 9,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2018-06-30T00:59:03",
  "op": [
    "comment",
    {
      "parent_author": "",
      "parent_permlink": "test",
      "author": "petecorey",
      "permlink": "test",
      "title": "this is a test",
      "body": "body",
      "json_metadata": ""
    }
  ]
}

{
  "trx_id": "6a99e52e4719d2688bb32e84e9c9399198e60641",
  "block": 20106582,
  "trx_in_block": 45,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2018-02-23T00:07:51",
  "op": [
    "account_update",
    {
      "account": "petecorey",
      "memo_key": "STM72cUgzYnCWx1MHgTdiv12U2bZeyJCCFMY67Rpd2sK72qhJdxJ2",
      "json_metadata": "{\"profile\":{\"profile_image\":\"http://www.east5th.co/img/portrait-150x150.png\",\"name\":\"Pete Corey\",\"website\":\"http://www.petecorey.com/blog/\"}}"
    }
  ]
}

{
  "trx_id": "1124c14ce2039bf2e608577eb3f0cb83e8d89e8c",
  "block": 20094534,
  "trx_in_block": 61,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2018-02-22T14:05:18",
  "op": [
    "comment",
    {
      "parent_author": "petecorey",
      "parent_permlink": "mining-for-bitcoin-vanity-addresses-with-elixir",
      "author": "cheetah",
      "permlink": "cheetah-re-petecoreymining-for-bitcoin-vanity-addresses-with-elixir",
      "title": "",
      "body": "Hi! I am a robot. I just upvoted you! I found similar content that readers might be interested in:\nhttp://www.petecorey.com/blog/2018/02/05/mining-for-bitcoin-vanity-addresses-with-elixir/",
      "json_metadata": ""
    }
  ]
}

{
  "trx_id": "393c1834363b74bfb15a42e422ee604653daf875",
  "block": 20094533,
  "trx_in_block": 29,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2018-02-22T14:05:15",
  "op": [
    "vote",
    {
      "voter": "cheetah",
      "author": "petecorey",
      "permlink": "mining-for-bitcoin-vanity-addresses-with-elixir",
      "weight": 8
    }
  ]
}

{
  "trx_id": "42f55c0b8e5c6ff88398550876c5e3e269d0cecd",
  "block": 20094529,
  "trx_in_block": 20,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2018-02-22T14:05:03",
  "op": [
    "comment",
    {
      "parent_author": "",
      "parent_permlink": "programming",
      "author": "petecorey",
      "permlink": "mining-for-bitcoin-vanity-addresses-with-elixir",
      "title": "Mining for Bitcoin Vanity Addresses with Elixir",
      "body": "<h1>Mining for Bitcoin Vanity Addresses with Elixir</h1>\n\n<p>We previously worked through the process of <a href=\"http://www.petecorey.com/blog/2018/01/22/generating-bitcoin-private-keys-and-public-addresses-with-elixir/\">generating a Bitcoin private address and translating it into a shareable public address</a> using only the tools and libraries shipped with Elixir and Erlang.</p>\n\n<p>The guiding force behind that article was Andreas Antonopoulos’ excellent <a href=\"http://amzn.to/2Eqcvi9\">Mastering Bitcoin</a> book.</p>\n\n<p>Let’s take another bite out of <a href=\"http://amzn.to/2Eqcvi9\">Mastering Bitcoin</a> and implement the algorithm Andreas describes for “mining for vanity addresses” at the end of chapter four. After we implement the basic algorithm, we’ll add our Elixir special sauce and turn it into a fully parallelized procedure.</p>\n\n<h2 id=\"what-is-a-vanity-address\">What is a Vanity Address?</h2>\n\n<p>The concept of a vanity address is simple. It’s a normal Bitcoin <a href=\"http://www.petecorey.com/blog/2018/01/22/generating-bitcoin-private-keys-and-public-addresses-with-elixir/#what-are-private-keys-and-public-addresses\">public address</a> that contains some sequence of desired characters.</p>\n\n<p>For example, a random Bitcoin public address might look like the following:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">1HKz4XU7ENT46ztEzsT83jRezyiDjvnBV8\n</code></pre></p>\n\n<p>On the live network, Bitcoin addresses always begin with <code class=\"language-elixir highlighter-rouge\">1</code>, but the remaining characters are entirely random.</p>\n\n<p>A vanity address might look like this:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">1pete7qrCiembh8AEf1zRP2zn6nDsLoHC\n</code></pre></p>\n\n<p>You’ll notice that the first five characters of this address are <code class=\"language-elixir highlighter-rouge\">1pete</code>. This isn’t an accident! I’ve intentionally sought out a public address that begins with my name, Pete, so people know who they’re sending their large sums of Bitcoin to.</p>\n\n<p>While the term “mining” sounds intimidating, the actual process of generating these vanity addresses is laughably simple.</p>\n\n<h2 id=\"how-do-you-mine-vanity-addresses\">How do you Mine Vanity Addresses?</h2>\n\n<p>“Mining,” in this context, is just another term for repeatedly doing something until some condition is met. As in, “keep digging until you find gold!”</p>\n\n<p>We’ll mine our vanity public address by repeatedly generating a private key, transforming it into a public address, and checking if the resulting address matches our desired pattern.</p>\n\n<p>That’s it!</p>\n\n<p>Building that in Elixir should be a walk in the park. We’ll start off by creating a new <code class=\"language-elixir highlighter-rouge\">VanityAddress</code> module and stubbing out a <code class=\"language-elixir highlighter-rouge\">generate_private_key/2</code> function:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">defmodule VanityAddress do\n  def generate_private_key(regex, version \\\\ &lt;&lt;0x00&gt;&gt;)\nend\n</code></pre></p>\n\n<p>Our <code class=\"language-elixir highlighter-rouge\">generate_private_key/2</code> function expects a <code class=\"language-elixir highlighter-rouge\">regex</code> which represents the pattern we’re trying to find in a vanity address (like <code class=\"language-elixir highlighter-rouge\">~r/^1pete/</code>), and a <code class=\"language-elixir highlighter-rouge\">version</code> byte that will used to indicate <a href=\"https://en.bitcoin.it/wiki/List_of_address_prefixes\">where this Bitcoin address will be used</a>.</p>\n\n<p>Within our <code class=\"language-elixir highlighter-rouge\">generate_private_key/2</code> function, we’ll kick off the mining process by generating a random private key and transforming it into a public address:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">private_key = PrivateKey.generate\npublic_address = PrivateKey.to_public_address(private_key)\n</code></pre></p>\n\n<p>If the <code class=\"language-elixir highlighter-rouge\">public_address</code> we generated matches the pattern we provided in our <code class=\"language-elixir highlighter-rouge\">regex</code>, we’ve successfully mined a vanity address! In that case, we’ll return the <code class=\"language-elixir highlighter-rouge\">private_key</code>. Otherwise, we’ll repeat the entire process with a recursive call to <code class=\"language-elixir highlighter-rouge\">generate_private_key/2</code>:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">case public_address =~ regex do\n  true -&gt; private_key\n  false -&gt; generate_private_key(regex, version)\nend\n</code></pre></p>\n\n<p>That’s all there is to it.</p>\n\n<p>We can use our new <code class=\"language-elixir highlighter-rouge\">generate_private_key/2</code> function in conjunction with the <code class=\"language-elixir highlighter-rouge\">PrivateKey.to_public_address/2</code> function we built last time to view our newly mined vanity key:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">VanityAddress.generate_private_key(~r/^1pete/)\n|&gt; PrivateKey.to_public_address\n</code></pre></p>\n\n<p>Congratulations; we’re miners!</p>\n\n<h2 id=\"thinking-in-parallel\">Thinking in Parallel</h2>\n\n<p>The problem with our simple implementation of <code class=\"language-elixir highlighter-rouge\">generate_private_key/2</code> is that <em>it’s slow</em>.</p>\n\n<p>While it’s true that the mining algorithm is inherently slow, there are many optimizations we could make to the code we’ve written. The most obvious improvement that comes to mind when using a “process-oriented” programming language like Elixir is to parallelize the mining algorithm across multiple processes.</p>\n\n<p>However, parallelizing our mining algorithm presents an interesting set of challenges.</p>\n\n<p>Each individual call to <code class=\"language-elixir highlighter-rouge\">generate_private_key/2</code> is completely synchronous and sequential. We won’t see much of a benefit by queuing up multiple concurrent calls to <code class=\"language-elixir highlighter-rouge\">generate_private_key/2</code> on the same CPU core. That said, while we’re running <code class=\"language-elixir highlighter-rouge\">generate_private_key/2</code> within a single process bound to a single CPU core, any other cores available to us are sitting idle.</p>\n\n<div style=\"width: 100%; margin: 4em 0;\">\n  <p><img src=\"https://s3-us-west-1.amazonaws.com/www.east5th.co/img/mining-for-bitcoin-vanity-addresses-with-elixir/sequential.png\" style=\"display: block; margin:1em auto; width: 60%;\" /></p>\n</div>\n\n<p>Ideally, we could simultaneously run as many instances of our <code class=\"language-elixir highlighter-rouge\">generate_private_key/2</code> execution as we have cores. The moment any of our parallel executions find a matching key, it would be returned to the caller.</p>\n\n<h2 id=\"creating-a-stream-of-parallel-tasks\">Creating a Stream of Parallel Tasks</h2>\n\n<p>Elixir’s little known (to me) <a href=\"https://hexdocs.pm/elixir/Task.html#async_stream/3\"><code class=\"language-elixir highlighter-rouge\">Task.async_stream/3</code> function</a> is the tool we need to implement this functionality.</p>\n\n<p><code class=\"language-elixir highlighter-rouge\">Task.async_stream/3</code> expects an enumerable as its first argument and a function to be applied concurrently to each element in the enumerable. Each element in the enumerable will have the provided function applied to it <em>in a new process.</em></p>\n\n<p>If we squint our eyes a little, we can see that this gives us what we need. The “enumerable” we pass into <code class=\"language-elixir highlighter-rouge\">Task.async_stream/3</code> will really be an infinite stream of zero-argument anonymous functions. Each of those anonymous functions simply calls <code class=\"language-elixir highlighter-rouge\">generate_private_key/2</code>.</p>\n\n<p>We’ll use <code class=\"language-elixir highlighter-rouge\">Stream.cycle/2</code> to create an infinite stream of these functions:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">[fn -&gt; generate_private_key(regex, version) end]\n|&gt; Stream.cycle\n</code></pre></p>\n\n<p>The function that we want to run in parallel simply executes each of those passed in anonymous functions, one at a time, each in its own process:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">|&gt; Task.async_stream(fn f -&gt; f.() end)\n</code></pre></p>\n\n<p>This is where our parallelization happens. Each call to <code class=\"language-elixir highlighter-rouge\">generate_private_key/2</code> is happening in a new process, and Elixir’s scheduler will spread each new process out over the available cores in the system.</p>\n\n<div style=\"width: 100%; margin: 4em 0;\">\n  <p><img src=\"https://s3-us-west-1.amazonaws.com/www.east5th.co/img/mining-for-bitcoin-vanity-addresses-with-elixir/parallel.png\" style=\"display: block; margin:1em auto; width: 60%;\" /></p>\n</div>\n\n<p>By default, <code class=\"language-elixir highlighter-rouge\">Task.async_stream/3</code> will run up to <code class=\"language-elixir highlighter-rouge\">System.schedulers_online/0</code> parallel instances of our <code class=\"language-elixir highlighter-rouge\">generate_private_key/2</code> execution, and <code class=\"language-elixir highlighter-rouge\">System.schedulers_online/0</code> defaults to <a href=\"https://stackoverflow.com/a/38701174/96048\">the number of available CPU cores in the system</a>. This means we’ll always have one instance of <code class=\"language-elixir highlighter-rouge\">generate_private_key/2</code> running on each of our cores.</p>\n\n<p>Perfect!</p>\n\n<h2 id=\"filtering-our-stream\">Filtering Our Stream</h2>\n\n<p><code class=\"language-elixir highlighter-rouge\">Task.async_stream/3</code> returns a stream that produces either an <code class=\"language-elixir highlighter-rouge\">{:ok, value}</code> tuple on success, or an <code class=\"language-elixir highlighter-rouge\">{:exit, reason}</code> tuple on failure. We don’t anticipate or care about failures in this situation, so we’ll <code class=\"language-elixir highlighter-rouge\">nil</code> them out with <code class=\"language-elixir highlighter-rouge\">Stream.map/2</code>:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">|&gt; Stream.map(fn\n  {:ok, thing} -&gt; thing\n  _ -&gt; nil\nend)\n</code></pre></p>\n\n<p>Now we can use <code class=\"language-elixir highlighter-rouge\">Stream.reject/2</code> to filter out any <code class=\"language-elixir highlighter-rouge\">nil</code> values from our mapped stream:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">|&gt; Stream.reject(&amp;(&amp;1 == nil))\n</code></pre></p>\n\n<p>Let’s wrap what we’ve done in a function called <code class=\"language-elixir highlighter-rouge\">stream_private_keys/2</code> that accepts a <code class=\"language-elixir highlighter-rouge\">regex</code> and a <code class=\"language-elixir highlighter-rouge\">version</code>:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">def stream_private_keys(regex, version \\\\ &lt;&lt;0x00&gt;&gt;) do\n  [fn -&gt; generate_private_key(regex, version) end]\n  |&gt; Stream.cycle\n  |&gt; Task.async_stream(fn f -&gt; f.() end)\n  |&gt; Stream.map(fn\n    {:ok, thing} -&gt; thing\n    _ -&gt; nil\n  end)\n  |&gt; Stream.reject(&amp;(&amp;1 == nil))\nend\n</code></pre></p>\n\n<p>What we’re left with is a stream that will produce any number of valid Bitcoin vanity addresses for a given <code class=\"language-elixir highlighter-rouge\">regex</code> and <code class=\"language-elixir highlighter-rouge\">version</code>, using all of the available CPU cores on our system.</p>\n\n<div style=\"width: 100%; margin: 4em 0;\">\n <p> <img src=\"https://s3-us-west-1.amazonaws.com/www.east5th.co/img/mining-for-bitcoin-vanity-addresses-with-elixir/stream.png\" style=\"display: block; margin:1em auto; width: 60%;\" /></p>\n</div>\n\n<h2 id=\"putting-our-stream-to-use\">Putting Our Stream to Use</h2>\n\n<p>Our stream <a href=\"https://hexdocs.pm/elixir/Stream.html#content\">doesn’t actually do anything</a> until we try to pull values out of it using a function from the <a href=\"https://hexdocs.pm/elixir/Enum.html#content\"><code class=\"language-elixir highlighter-rouge\">Enum</code> module</a>. Let’s use <code class=\"language-elixir highlighter-rouge\">Enum.take/2</code> to pull out three vanity Bitcoin addresses that match our desired pattern (<code class=\"language-elixir highlighter-rouge\">123</code>):</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">VanityAddress.stream_private_keys(~r/^123/)\n|&gt; Enum.take(3)\n|&gt; Enum.map(&amp;PrivateKey.to_public_address/1)\n</code></pre></p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">[\"123avbA76Zk98jik3ymTHkjbjKKftAhJiZ\",\n \"123aGknGk5F2NPkB6e4e6pehVGL7gBR8az\",\n \"123hzDB1CxcyDwusfsb8Pfh3Ti2i4NQLGR\"]\n</code></pre></p>\n\n<p>If we take a look at our CPU usage while our mining pipeline is chugging away, we’ll see that all of the CPUs on our machine are being fully utilized.</p>\n\n<p>Success!</p>\n\n<h2 id=\"final-thoughts\">Final Thoughts</h2>\n\n<p>Spoiler alert: the process of mining for Bitcoin is nearly identical to mining for vanity addresses. Instead of hashing private keys and looking for a random leading string like <code class=\"language-elixir highlighter-rouge\">1pete</code>, Bitcoin miners hash transaction data, looking for hashes that begin with some number of leading zeros corresponding to <a href=\"https://en.bitcoin.it/wiki/Difficulty\">the current block difficulty</a>.</p>\n\n<p>There’s a huge amount of pomp and circumstance around the term “mining”, but at its core, it’s an incredibly simple and approachable idea.</p>\n\n<p>Be sure to check out <a href=\"https://github.com/pcorey/hello_bitcoin/blob/master/lib/vanity_address.ex\">the <code class=\"language-elixir highlighter-rouge\">VanityAddress</code> module</a> in <a href=\"https://github.com/pcorey/hello_bitcoin\">my <code class=\"language-elixir highlighter-rouge\">hello_bitcoin</code> project</a> on Github, and if this kind of thing is at all interesting to you, I highly recommend you pick up a copy of Andreas Antonopoulos’ <a href=\"http://amzn.to/2Eqcvi9\">Mastering Bitcoin</a>.</p>\n\n<p><em>Originally posted on February 5, 2018 at <a href=\"http://www.petecorey.com/blog/2018/02/05/mining-for-bitcoin-vanity-addresses-with-elixir/\" >www.petecorey.com</a>.</em></p>",
      "json_metadata": "{\"tags\":[\"programming\",\"bitcoin\",\"elixir\"],\"image\":[\"https://s3-us-west-1.amazonaws.com/www.east5th.co/img/mining-for-bitcoin-vanity-addresses-with-elixir/sequential.png\",\"https://s3-us-west-1.amazonaws.com/www.east5th.co/img/mining-for-bitcoin-vanity-addresses-with-elixir/parallel.png\",\"https://s3-us-west-1.amazonaws.com/www.east5th.co/img/mining-for-bitcoin-vanity-addresses-with-elixir/stream.png\"],\"links\":[\"http://www.petecorey.com/blog/2018/01/22/generating-bitcoin-private-keys-and-public-addresses-with-elixir/\",\"http://amzn.to/2Eqcvi9\",\"http://www.petecorey.com/blog/2018/01/22/generating-bitcoin-private-keys-and-public-addresses-with-elixir/#what-are-private-keys-and-public-addresses\",\"https://en.bitcoin.it/wiki/List_of_address_prefixes\",\"https://hexdocs.pm/elixir/Task.html#async_stream/3\",\"https://stackoverflow.com/a/38701174/96048\",\"https://hexdocs.pm/elixir/Stream.html#content\",\"https://hexdocs.pm/elixir/Enum.html#content\",\"https://en.bitcoin.it/wiki/Difficulty\",\"https://github.com/pcorey/hello_bitcoin/blob/master/lib/vanity_address.ex\",\"https://github.com/pcorey/hello_bitcoin\",\"http://www.petecorey.com/blog/2018/02/05/mining-for-bitcoin-vanity-addresses-with-elixir/\"],\"app\":\"steemit/0.1\",\"format\":\"markdown\"}"
    }
  ]
}

{
  "trx_id": "1d15dedaf37eb3943c310371386f6894b7692bd3",
  "block": 20094397,
  "trx_in_block": 28,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2018-02-22T13:58:27",
  "op": [
    "comment",
    {
      "parent_author": "petecorey",
      "parent_permlink": "generating-bitcoin-private-keys-and-public-addresses-with-elixir",
      "author": "cheetah",
      "permlink": "cheetah-re-petecoreygenerating-bitcoin-private-keys-and-public-addresses-with-elixir",
      "title": "",
      "body": "Hi! I am a robot. I just upvoted you! I found similar content that readers might be interested in:\nhttp://www.petecorey.com/blog/2018/01/22/generating-bitcoin-private-keys-and-public-addresses-with-elixir/",
      "json_metadata": ""
    }
  ]
}

{
  "trx_id": "118871279c566512e5f4a49167b9a16ae15ef79f",
  "block": 20094396,
  "trx_in_block": 23,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2018-02-22T13:58:24",
  "op": [
    "vote",
    {
      "voter": "cheetah",
      "author": "petecorey",
      "permlink": "generating-bitcoin-private-keys-and-public-addresses-with-elixir",
      "weight": 8
    }
  ]
}

{
  "trx_id": "9bec1e7d32d889b880352390937d7287797f1451",
  "block": 20094377,
  "trx_in_block": 5,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2018-02-22T13:57:27",
  "op": [
    "comment",
    {
      "parent_author": "",
      "parent_permlink": "programming",
      "author": "petecorey",
      "permlink": "generating-bitcoin-private-keys-and-public-addresses-with-elixir",
      "title": "Generating Bitcoin Private Keys and Public Addresses with Elixir",
      "body": "<h1>Generating Bitcoin Private Keys and Public Addresses with Elixir</h1>\n\n<p>Lately I’ve been working my way through <a href=\"http://amzn.to/2DAbVy0\">Mastering Bitcoin</a>, implementing as many of the examples in the book in Elixir as I can.</p>\n\n<p>I’ve been amazed at how well Elixir has fared with implementing the algorithms involved in working with Bitcoin keys and addresses. Elixir ships with all the tools required to generate a cryptographically secure private key and transform it into a public address string.</p>\n\n<p>Let’s walk through the process step by step and build our our own Elixir module to generate private keys and public addresses.</p>\n\n<h2 id=\"what-are-private-keys-and-public-addresses\">What are Private Keys and Public Addresses?</h2>\n\n<p>A Bitcoin private key is really just a random two hundred fifty six bit number. As the name implies, this number is intended to be kept private.</p>\n\n<p>From each private key, a public-facing Bitcoin address can be generated. Bitcoin can be sent to this public address by anyone in the world. However, only the keeper of the private key can produce a signature that allows them to access the Bitcoin stored there.</p>\n\n<p>Let’s use Elixir to generate a cryptographically secure private key and then generate its most basic corresponding public address so we can receive some Bitcoin!</p>\n\n<h2 id=\"pulling-a-private-key-out-of-thin-air\">Pulling a Private Key Out of Thin Air</h2>\n\n<p>As I mentioned earlier, a Bitcoin private key is really just a random two hundred and fifty six bit number. In other words, a private key can be any number between <code class=\"language-elixir highlighter-rouge\">0</code> and <code class=\"language-elixir highlighter-rouge\">2^256</code>.</p>\n\n<p>However, not all random numbers are created equally. We need to be sure that we’re generating our random number from a <a href=\"https://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator\">cryptographically secure source of entropy</a>. Thankfully, Elixir exposes <a href=\"http://erlang.org/doc/man/crypto.html#strong_rand_bytes-1\">Erlang’s <code class=\"language-elixir highlighter-rouge\">:crypto.strong_rand_bytes/1</code></a> function which lets us easily generate a list of truly random bytes.</p>\n\n<p>Let’s use <code class=\"language-elixir highlighter-rouge\">:crypto.strong_rand_bytes/1</code> as the basis for our private key generator. We’ll start by creating a new <code class=\"language-elixir highlighter-rouge\">PrivateKey</code> module and a <code class=\"language-elixir highlighter-rouge\">generate/0</code> function that takes no arguments:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">defmodule PrivateKey do\n  def generate\nend\n</code></pre></p>\n\n<p>Inside our <code class=\"language-elixir highlighter-rouge\">generate/0</code> function, we’ll request <code class=\"language-elixir highlighter-rouge\">32</code> random bytes (or <code class=\"language-elixir highlighter-rouge\">256</code> bits) from <code class=\"language-elixir highlighter-rouge\">:crypto.strong_rand_bytes/1</code>:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">def generate do\n  :crypto.strong_rand_bytes(32)\nend\n</code></pre></p>\n\n<p>This gives us a random set of <code class=\"language-elixir highlighter-rouge\">32</code> bytes that, when viewed as an unsigned integer, ranges between <code class=\"language-elixir highlighter-rouge\">0</code> and <code class=\"language-elixir highlighter-rouge\">2^256 - 1</code>.</p>\n\n<p>Unfortunately, we’re not quite done.</p>\n\n<h2 id=\"validating-our-private-key\">Validating our Private Key</h2>\n\n<p>To ensure that our private key is difficult to guess, the <a href=\"http://www.secg.org/\">Standards for Efficient Cryptography Group</a> recommends that we pick a private key between the number <code class=\"language-elixir highlighter-rouge\">1</code> and a number slightly smaller than <code class=\"language-elixir highlighter-rouge\">1.158e77</code>:</p>\n\n<p style=\"text-align: center\"><a href=\"https://s3-us-west-1.amazonaws.com/www.east5th.co/img/secg.png\" style=\"display: block; background-color: transparent; color: #ccc; text-align: center; line-height: 1; font-size: 0.8; margin: 2em auto;\"><img style=\"display:block; width: 75%; margin: 0 auto 1em;\" src=\"https://s3-us-west-1.amazonaws.com/www.east5th.co/img/secg.png\" />An excerpt of the SECG guidelines.</a></p>\n\n<p>We can add this validation check fairly easily by adding the SECG-provided upper bound as an attribute to our <code class=\"language-elixir highlighter-rouge\">PrivateKey</code> module:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">@n :binary.decode_unsigned(&lt;&lt;\n  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,\n  0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFE,\n  0xBA, 0xAE, 0xDC, 0xE6, 0xAF, 0x48, 0xA0, 0x3B,\n  0xBF, 0xD2, 0x5E, 0x8C, 0xD0, 0x36, 0x41, 0x41\n&gt;&gt;)\n</code></pre></p>\n\n<p>Next, we’ll add a <code class=\"language-elixir highlighter-rouge\">valid?/1</code> function to our module that returns <code class=\"language-elixir highlighter-rouge\">true</code> if the provided secret key falls within this range, and <code class=\"language-elixir highlighter-rouge\">false</code> if it does not:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">defp valid?(key) when key &gt; 1 and key &lt; @n, do: true\ndefp valid?(_), do: false\n</code></pre></p>\n\n<p>Before we pass our private key into our <code class=\"language-elixir highlighter-rouge\">valid?/1</code> function, we’ll need to convert it from a thirty two byte binary into an unsigned integer. Let’s add a third <code class=\"langauge-elixir highlighter-rouge\">valid?/1</code> function head that does just that:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">defp valid?(key) when is_binary(key) do\n  key\n  |&gt; :binary.decode_unsigned\n  |&gt; valid?\nend\n</code></pre></p>\n\n<p>We’ll finish off our validation by passing our generated private key into our new <code class=\"language-elixir highlighter-rouge\">valid?/1</code> function. If the key is valid, we’ll return it. Otherwise, we’ll generate a new private key and try again:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">def generate do\n  private_key = :crypto.strong_rand_bytes(32)\n  case valid?(private_key) do\n    true  -&gt; private_key\n    false -&gt; generate\n  end\nend\n</code></pre></p>\n\n<p>Now we can call <code class=\"language-elixir highlighter-rouge\">PrivateKey.generate</code> to generate a new Bitcoin private key!</p>\n\n<h2 id=\"from-private-key-to-public-key-\">From Private Key to Public Key …</h2>\n\n<p>The most basic process for turning a Bitcoin private key into a sharable public address involves three basic steps. The first step is to transform our private key into a public key with the help of <a href=\"https://en.wikipedia.org/wiki/Elliptic-curve_cryptography\">elliptic curve cryptography</a>.</p>\n\n<p>We’ll start by adding a new <code class=\"language-elixir highlighter-rouge\">to_public_key/1</code> function to our <code class=\"language-elixir highlighter-rouge\">PrivateKey</code> module:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">def to_public_key(private_key)\n</code></pre></p>\n\n<p>In our <code class=\"language-elixir highlighter-rouge\">to_public_key/1</code> function, we’ll use Erlang’s <code class=\"language-elixir highlighter-rouge\">:crypto.generate_key</code> function to sign our <code class=\"language-elixir highlighter-rouge\">private_key</code> using an elliptic curve. We’ll specifically use <a href=\"https://en.bitcoin.it/wiki/Secp256k1\">the <code class=\"language-elixir highlighter-rouge\">:secp256k1</code> curve</a>:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">:crypto.generate_key(:ecdh, :crypto.ec_curve(:secp256k1), private_key)\n</code></pre></p>\n\n<p>We’re using the elliptic curve key generation as <a href=\"https://en.wikipedia.org/wiki/Trapdoor_function\">a trapdoor function</a> to ensure our private key’s secrecy. It’s easy for us to generate our public key from our private key, but reversing the computation and generating our private key from our public key is nearly impossible.</p>\n\n<p>The <code class=\"language-elixir highlighter-rouge\">:crypto.generate_key</code> function returns a two-element tuple. The first element in this tuple is our Bitcoin public key. We’ll pull it out using Elixir’s <code class=\"language-elixir highlighter-rouge\">elem/1</code> function:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">:crypto.generate_key(:ecdh, :crypto.ec_curve(:secp256k1), private_key)\n|&gt; elem(0)\n</code></pre></p>\n\n<p>The returned value is a sixty five byte binary representing our public key!</p>\n\n<h2 id=\"-public-key-to-public-hash-\">… Public Key to Public Hash …</h2>\n\n<p>Once we have our public key in memory, our next step in transforming it into a public address is to hash it. This gives us what’s called the “public hash” of our public key.</p>\n\n<p>Let’s make a new function, <code class=\"language-elixir highlighter-rouge\">to_public_hash/1</code> that takes our <code class=\"language-elixir highlighter-rouge\">private_key</code> as an argument:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">def to_public_hash(private_key)\n</code></pre></p>\n\n<p>We’ll start the hashing process by turning our <code class=\"language-elixir highlighter-rouge\">private_key</code> into a public key with a call to <code class=\"language-elixir highlighter-rouge\">to_public_key</code>:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">private_key\n|&gt; to_public_key\n</code></pre></p>\n\n<p>Next, we pipe our public key through two hashing functions: <a href=\"https://en.wikipedia.org/wiki/SHA-2\">SHA-256</a>, followed by <a href=\"https://en.wikipedia.org/wiki/RIPEMD\">RIPEMD-160</a>:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">private_key\n|&gt; to_public_key\n|&gt; hash(:sha256)\n|&gt; hash(:ripemd160)\n</code></pre></p>\n\n<p>Bitcoin uses the RIPEMD-160 hashing algorithm because it produces a short hash. The intermediate SHA-256 hashing is used <a href=\"https://bitcoin.stackexchange.com/a/9216\">to prevent insecurities through unexpected interactions</a> between our elliptic curve signing algorithm and the RIPEMD algorithm.</p>\n\n<p>In this example, <code class=\"language-elixir highlighter-rouge\">hash/1</code> is a helper function that wraps Erlang’s <code class=\"language-elixir highlighter-rouge\">:crypto.hash</code>.</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">defp hash(data, algorithm), do: :crypto.hash(algorithm, data)\n</code></pre></p>\n\n<p>Flipping the arguments to <code class=\"language-elixir highlighter-rouge\">:crypto.hash</code> in this way lets us easily pipe our data through the <code class=\"language-elixir highlighter-rouge\">hash/1</code> helper.</p>\n\n<h2 id=\"-and-public-hash-to-public-address\">… And Public Hash to Public Address</h2>\n\n<p>Lastly, we can convert our public hash into a full-fledged Bitcoin address by <a href=\"http://www.petecorey.com/blog/2018/01/08/bitcoins-base58check-in-pure-elixir/\">Base58Check encoding</a> the hash with a version byte corresponding to <a href=\"https://en.bitcoin.it/wiki/List_of_address_prefixes\">the network where we’re using the address</a>.</p>\n\n<p>Let’s add a <code class=\"language-elixir highlighter-rouge\">to_public_address/2</code> function to our <code class=\"language-elixir highlighter-rouge\">PrivateKey</code> module:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">def to_public_address(private_key, version \\\\ &lt;&lt;0x00&gt;&gt;)\n</code></pre></p>\n\n<p>The <code class=\"language-elixir highlighter-rouge\">to_public_address/2</code> function takes a <code class=\"language-elixir highlighter-rouge\">private_key</code> and a <code class=\"language-elixir highlighter-rouge\">version</code> byte as its arguments. The <code class=\"langauge-elixir highlighter-rouge\">version</code> defaults to <code class=\"langauge-elixir highlighter-rouge\">&lt;&lt;0x00&gt;&gt;</code>, indicating that this address will be used on the live Bitcoin network.</p>\n\n<p>To create a Bitcoin address, we start by converting our <code class=\"language-elixir highlighter-rouge\">private_key</code> into a public hash with a call to <code class=\"language-elixir highlighter-rouge\">to_public_hash/1</code>:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">private_key\n|&gt; to_public_hash\n</code></pre></p>\n\n<p>All that’s left to do is Base58Check encode the resulting hash with the provided <code class=\"language-elixir highlighter-rouge\">version</code> byte:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">private_key\n|&gt; to_public_hash\n|&gt; Base58Check.encode(version)\n</code></pre></p>\n\n<p>After laying the groundwork, the final pieces of the puzzle effortlessly fall into place.</p>\n\n<h2 id=\"putting-our-creation-to-use\">Putting Our Creation to Use</h2>\n\n<p>Now that we can generate cryptographically secure private keys and transform them into publishable public addresses, we’re in business.</p>\n\n<p>Literally!</p>\n\n<p>Let’s generate a new private key, transform it into its corresponding public address, and try out on <a href=\"https://en.bitcoin.it/wiki/Testnet\">the Bitcoin testnet</a>. We’ll start by generating our private key:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">private_key = PrivateKey.generate\n</code></pre></p>\n\n<p>This gives us a thirty two byte binary. If we wanted, we could Base58Check encode this with <a href=\"https://en.bitcoin.it/wiki/List_of_address_prefixes\">a testnet <code class=\"language-elixir highlighter-rouge\">version</code> byte of <code class=\"language-elixir highlighter-rouge\">0xEF</code></a>. This is known as the “Wallet Import Format”, or WIF, of our Bitcoin private key:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">Base58Check.encode(private_key, &lt;&lt;0xEF&gt;&gt;)\n</code></pre></p>\n\n<p>As its name suggests, converting our private key into a WIF allows us to easily import it into most Bitcoin wallet software:</p>\n\n<p><a href=\"https://s3-us-west-1.amazonaws.com/www.east5th.co/img/electrum-import.png\" style=\"display: block; background-color: transparent; color: #ccc; text-align: center; line-height: 1; font-size: 0.8; margin: 2em auto;\"><img style=\"display:block; width: 75%; margin: 0 auto 1em;\" src=\"https://s3-us-west-1.amazonaws.com/www.east5th.co/img/electrum-import.png\" />Importing our test private key.</a></p>\n\n<p>Next, let’s convert our private key into a testnet public address using <a href=\"https://en.bitcoin.it/wiki/List_of_address_prefixes\">a <code class=\"language-elixir highlighter-rouge\">version</code> byte of <code class=\"language-elixir highlighter-rouge\">0x6F</code></a>:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">PrivateKey.to_public_address(private_key, &lt;&lt;0x6F&gt;&gt;)\n</code></pre></p>\n\n<p>Now that we have our public address, let’s find <a href=\"https://testnet.manu.backend.hamburg/faucet\">a testnet faucet</a> and send a few tBTC to our newly generated address! After initiating the transaction with our faucet, we should see our Bitcoin arrive at our address on either <a href=\"https://www.blocktrail.com/tBTC/tx/d75b1080a0ad2343c6ad89d35a465d18a0c59a5848cfd773814792d19a4afd48\">a blockchain explorer</a>, or within our wallet software.</p>\n\n<p><a href=\"https://s3-us-west-1.amazonaws.com/www.east5th.co/img/electrum-confirm.png\" style=\"display: block; background-color: transparent; color: #ccc; text-align: center; line-height: 1; font-size: 0.8; margin: 2em auto;\"><img style=\"display:block; width: 75%; margin: 0 auto 1em;\" src=\"https://s3-us-west-1.amazonaws.com/www.east5th.co/img/electrum-confirm.png\" />Our tBTC has arrived.</a></p>\n\n<p>Victory!</p>\n\n<h2 id=\"final-thoughts\">Final Thoughts</h2>\n\n<p>Elixir, thanks to its Erlang heritage, ships with a wealth of tools that make this kind of hashing, signing, and byte mashing a walk in the park.</p>\n\n<p>I encourage you to check our <a href=\"https://github.com/pcorey/hello_bitcoin/blob/master/lib/private_key.ex\">the <code class=\"language-elixir highlighter-rouge\">PrivateKey</code> module on Github</a> to get a better feel for the simplicity of the code we wrote today. Overall, I’m very happy with the result.</p>\n\n<p>If you enjoyed this article, I highly recommend you check out <a href=\"http://amzn.to/2D5ARfK\">the Mastering Bitcoin book</a>. If you <em>really enjoyed</em> this article, feel free to send a few Bitcoin to this address I generated using our new <code class=\"language-elixir highlighter-rouge\">PrivateKey</code> module:</p>\n\n<p><pre class=\"language-elixir\"><code class=\"language-elixir\">1HKz4XU7ENT46ztEzsT83jRezyiDjvnBV8\n</code></pre></p>\n\n<p>Stay tuned for more Bitcoin-related content as I work my way through <a href=\"http://amzn.to/2D5ARfK\">Mastering Bitcoin</a>!</p>",
      "json_metadata": "{\"tags\":[\"programming\",\"bitcoin\",\"elixir\"],\"image\":[\"https://s3-us-west-1.amazonaws.com/www.east5th.co/img/secg.png\",\"https://s3-us-west-1.amazonaws.com/www.east5th.co/img/electrum-import.png\",\"https://s3-us-west-1.amazonaws.com/www.east5th.co/img/electrum-confirm.png\"],\"links\":[\"http://amzn.to/2DAbVy0\",\"https://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator\",\"http://erlang.org/doc/man/crypto.html#strong_rand_bytes-1\",\"http://www.secg.org/\",\"https://s3-us-west-1.amazonaws.com/www.east5th.co/img/secg.png\",\"https://en.wikipedia.org/wiki/Elliptic-curve_cryptography\",\"https://en.bitcoin.it/wiki/Secp256k1\",\"https://en.wikipedia.org/wiki/Trapdoor_function\",\"https://en.wikipedia.org/wiki/SHA-2\",\"https://en.wikipedia.org/wiki/RIPEMD\",\"https://bitcoin.stackexchange.com/a/9216\",\"http://www.petecorey.com/blog/2018/01/08/bitcoins-base58check-in-pure-elixir/\",\"https://en.bitcoin.it/wiki/List_of_address_prefixes\",\"https://en.bitcoin.it/wiki/Testnet\",\"https://s3-us-west-1.amazonaws.com/www.east5th.co/img/electrum-import.png\",\"https://testnet.manu.backend.hamburg/faucet\",\"https://www.blocktrail.com/tBTC/tx/d75b1080a0ad2343c6ad89d35a465d18a0c59a5848cfd773814792d19a4afd48\",\"https://s3-us-west-1.amazonaws.com/www.east5th.co/img/electrum-confirm.png\",\"https://github.com/pcorey/hello_bitcoin/blob/master/lib/private_key.ex\",\"http://amzn.to/2D5ARfK\"],\"app\":\"steemit/0.1\",\"format\":\"markdown\"}"
    }
  ]
}

{
  "trx_id": "149290a7a329d450af32b90f6c686dc788aa9ce0",
  "block": 19730345,
  "trx_in_block": 14,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2018-02-09T22:24:33",
  "op": [
    "comment",
    {
      "parent_author": "petecorey",
      "parent_permlink": "bitcoin-s-base58check-in-pure-elixir",
      "author": "cheetah",
      "permlink": "cheetah-re-petecoreybitcoin-s-base58check-in-pure-elixir",
      "title": "",
      "body": "Hi! I am a robot. I just upvoted you! I found similar content that readers might be interested in:\nhttp://www.petecorey.com/blog/2018/01/08/bitcoins-base58check-in-pure-elixir/",
      "json_metadata": ""
    }
  ]
}

{
  "trx_id": "9f568d43516b7f173ba4f92811cabcd7b5b9b56c",
  "block": 19730343,
  "trx_in_block": 31,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2018-02-09T22:24:27",
  "op": [
    "vote",
    {
      "voter": "cheetah",
      "author": "petecorey",
      "permlink": "bitcoin-s-base58check-in-pure-elixir",
      "weight": 8
    }
  ]
}

{
  "trx_id": "9de1db9476ac63673f31439f1aa1746083793b11",
  "block": 19730338,
  "trx_in_block": 14,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2018-02-09T22:24:12",
  "op": [
    "comment",
    {
      "parent_author": "",
      "parent_permlink": "programming",
      "author": "petecorey",
      "permlink": "bitcoin-s-base58check-in-pure-elixir",
      "title": "Bitcoin's Base58Check in Pure Elixir",
      "body": "<html>\n<h1>Bitcoin's Base58Check in Pure Elixir</h1>\n<p>An important piece of the process of transforming a Bitcoin private key into a public address, as outlined in <a href=\"http://amzn.to/2E6gO0I\">the fantastic Mastering Bitcoin book</a>, is the Base58Check encoding algorithm.</p>\n<p>The Bitcoin wiki has a <a href=\"https://en.bitcoin.it/wiki/Base58Check_encoding\">great article</a> on Base58Check encoding, and even gives an <a href=\"https://en.bitcoin.it/wiki/Base58Check_encoding#Base58_symbol_chart\">example implementation</a> of the underlying Base58 encoding algorithm in C.</p>\n<p>This algorithm seems especially well-suited to Elixir, so I thought it’d be a fun and useful exercise to build out <code>Base58</code> and <code>Base58Check</code> modules to use in future Bitcoin and Elixir experiments.</p>\n<h2>Like Base64, but Less Confusing</h2>\n<p>Base58 is a binary-to-text encoding algorithm that’s designed to encode a blob of arbitrary binary data into human readable text, much like the more well known <a href=\"https://hexdocs.pm/elixir/Base.html#encode64/2\">Base64 algorithm</a>.</p>\n<p>Unlike Base64 encoding, Bitcoin’s Base58 encoding algorithm omits characters that can be potentially confusing or ambiguous to a human reader. For example, the characters <code>O</code> and <code>0</code>, or <code>I</code> and <code>l</code> can look similar or identical to some readers or users of certain fonts.</p>\n<p>To avoid that ambiguity, Base58 simply removes those characters from its alphabet.</p>\n<p>Shrinking the length of the alphabet we map our binary data onto from sixty four characters down to fifty eight characters means that we can’t simply <a href=\"https://en.wikipedia.org/wiki/Base64#Examples\">group our binary into six-bit chunks</a> and map each chunk onto its corresponding letter in our alphabet.</p>\n<p>Instead, our Base58 encoding algorithm works by treating our binary as a single large number. We repeatedly divide that number by the size of our alphabet (fifty eight), and use the remainder of that division to map onto a character in our alphabet.</p>\n<h2>Implementing Base58 in Elixir</h2>\n<p>This kind of algorithm can neatly be expressed in Elixir. We’ll start by creating a <code>Base58</code> module and adding our alphabet as a module attribute:</p>\n<pre><code>defmodule Base58 do<br>\n &nbsp;@alphabet '123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz'<br>\nend</code></pre>\n<p>Inside our <code>Base58</code> module, we’ll define an <code>encode/2</code> function. If we pass <code>encode</code> a binary, we want to convert it into a number using <a href=\"http://erlang.org/doc/man/binary.html#decode_unsigned-1\">Erlang’s <code>:binary.decode_unsigned</code></a>:</p>\n<pre><code>def encode(data, hash \\\\ \"\")<br>\ndef encode(data, hash) when is_binary(data) do<br>\n &nbsp;encode(:binary.decode_unsigned(data), hash)<br>\nend</code></pre>\n<p>Once converted, we pass our binary-come-number into a recursive call to <code>encode/2</code> along with the beginning of our hash, an empty string.</p>\n<p>For each recursive call to <code>encode/2</code>, we use <code>div</code> and <code>rem</code> to divide our number by <code>58</code> and find the reminder. We use that remainder to map into our <code>@alphabet</code>, and prepend the resulting character onto our <code>hash</code>:</p>\n<pre><code>def encode(data, hash) do<br>\n &nbsp;character = &lt;&lt;Enum.at(@alphabet, rem(data, 58))&gt;&gt;<br>\n &nbsp;encode(div(data, 58), hash &lt;&gt; character)<br>\nend</code></pre>\n<p>We’ll continue recursing until we’ve divided our <code>data</code> down to <code>0</code>. In that case, we’ll return the <code>hash</code> string we’ve built up:</p>\n<pre><code>def encode(0, hash), do: hash</code></pre>\n<p>This implementation of our Base58 encoded <em>mostly</em> works. We can encode any text string and receive correct results:</p>\n<pre><code>iex(1)&gt; Base58.encode(\"hello\")<br>\n\"Cn8eVZg\"</code></pre>\n<h2>Encoding Leading Zeros</h2>\n<p>However when we try to encode binaries with leading zero bytes, those bytes vanish from our resulting hash:</p>\n<pre><code>iex(1)&gt; Base58.encode(&lt;&lt;0x00&gt;&gt; &lt;&gt; \"hello\")<br>\n\"Cn8eVZg\"</code></pre>\n<p>That zero <em>should</em> become a leading <code>\"1\"</code> in our resulting hash, but our process of converting the initial binary into a number is truncating those leading bytes. We’ll need to count those leading zeros, encode them manually, and prepend them to our final hash.</p>\n<p>Let’s start by writing a function that counts the number of leading zeros in our initial binary:</p>\n<pre><code>defp leading_zeros(data) do<br>\n &nbsp;:binary.bin_to_list(data)<br>\n &nbsp;|&gt; Enum.find_index(&amp;(&amp;1 != 0))<br>\nend</code></pre>\n<p>We use <a href=\"http://erlang.org/doc/man/erlang.html#binary_to_list-1\">Erlang’s <code>:binary.bin_to_list</code></a> to convert our binary into a list of bytes, and <a href=\"https://hexdocs.pm/elixir/Enum.html#find_index/2\"><code>Enum.find_index</code></a>to find the first byte in our list that isn’t zero. This index value is equivalent to the number of leading zero bytes in our binary.</p>\n<p>Next, we’ll write a function to manually encode those leading zeros:</p>\n<pre><code>defp encode_zeros(data) do<br>\n &nbsp;&lt;&lt;Enum.at(@alphabet, 0)&gt;&gt;<br>\n &nbsp;|&gt; String.duplicate(leading_zeros(data)) <br>\nend</code></pre>\n<p>We simply grab the character in our alphabet that maps to a zero byte (<code>\"1\"</code>), and duplicate it as many times as we need.</p>\n<p>Finally, we’ll update our initial <code>encode/2</code> function to prepend these leading zeros onto our resulting hash:</p>\n<pre><code>def encode(data, hash) when is_binary(data) do<br>\n &nbsp;encode_zeros(data) &lt;&gt; encode(:binary.decode_unsigned(data), hash)<br>\nend</code></pre>\n<p>Now we should be able to encode binaries with leading zero bytes and see their resulting <code>\"1\"</code> values in our final hash:</p>\n<pre><code>iex(1)&gt; Base58.encode(&lt;&lt;0x00&gt;&gt; &lt;&gt; \"hello\")<br>\n\"1Cn8eVZg\"</code></pre>\n<p>Great!</p>\n<h2>Base58 + Checksum = Base58Check</h2>\n<p>Now that we have a working implementation of the Base58 encoding algorithm, we can implement our Base58Check algorithm!</p>\n<p>Base58Check encoding is really just Base58 with an added checksum. This checksum is important to in the Bitcoin world to ensure that public addresses aren’t mistyped or corrupted before funds are exchanged.</p>\n<p>At a high level, the process of Base58Check encoding a blob of binary data involves hashing that data, taking the first four bytes of the resulting hash and appending them to the end of the binary, and Base58 encoding the result.</p>\n<p>We can implement Base58Check fairly easily using our newly written <code>Base58</code> module. We’ll start by creating a new <code>Base58Check</code> module:</p>\n<pre><code>defmodule Base58Check do<br>\nend</code></pre>\n<p>In our module, we’ll define a new <code>encode/2</code> function that takes a version byte and the binary we want to encode:</p>\n<pre><code>def encode(version, data)</code></pre>\n<p>Bitcoin uses the <code>version</code> byte to specify the type of address being encoded. A version byte of <code>0x00</code>means that we’re encoding a regular Bitcoin address to be used on the live Bitcoin network.</p>\n<p>The first thing we’ll need to do is generate our checksum from our <code>version</code> and our <code>data</code>. We’ll do that in a new function:</p>\n<pre><code>defp checksum(version, data) do<br>\n &nbsp;version &lt;&gt; data<br>\n &nbsp;|&gt; sha256<br>\n &nbsp;|&gt; sha256<br>\n &nbsp;|&gt; split<br>\nend</code></pre>\n<p>We concatenate our <code>version</code> and <code>data</code> binaries together, hash them twice using a <code>sha256/1</code> helper function, and then returning the first four bytes of the resulting hash with a call to <code>split/1</code>.</p>\n<p><code>split/1</code> is a helper function that pulls the first four bytes out of the resulting hash using binary pattern matching:</p>\n<pre><code>defp split(&lt;&lt; hash :: bytes-size(4), _ :: bits &gt;&gt;), do: hash</code></pre>\n<p>Our <code>sha256/1</code> helper function uses <a href=\"http://erlang.org/doc/man/crypto.html#hash-2\">Erlang’s <code>:crypto.hash</code></a> function to <a href=\"https://en.wikipedia.org/wiki/SHA-2\">SHA-256 hash</a> its argument:</p>\n<pre><code>defp sha256(data), do: :crypto.hash(:sha256, data)</code></pre>\n<p>We’ve wrapped this in a helper function to facilitate Elixir-style piping.</p>\n<p><br></p>\n<p>Now that we have our four-byte checksum, we can flesh out our original <code>encode/2</code> function:</p>\n<pre><code>def encode(version, data) do<br>\n &nbsp;version &lt;&gt; data &lt;&gt; checksum(version, data)<br>\n &nbsp;|&gt; Base58.encode<br>\nend</code></pre>\n<p>We concatenate our <code>version</code>, <code>data</code>, and the result of our <code>checksum</code> function together, and Base58 encode the result. That’s it!</p>\n<p>Base58Check encoding our <code>\"hello\"</code> string with a <code>version</code> of <code>&lt;&lt;0x00&gt;&gt;</code> should give us a result of <code>\"12L5B5yqsf7vwb\"</code>. We can go further and <a href=\"https://en.bitcoin.it/wiki/Technical_background_of_version_1_Bitcoin_addresses\">verify our implementation with an example pulled from the Bitcoin wiki</a>:</p>\n<pre><code>iex(1)&gt; Base58Check.encode(&lt;&lt;0x00&gt;&gt;, <br>\n &nbsp;&nbsp;&nbsp;&lt;&lt;0x01, 0x09, 0x66, 0x77, 0x60,<br>\n &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0x06, 0x95, 0x3D, 0x55, 0x67,<br>\n &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0x43, 0x9E, 0x5E, 0x39, 0xF8, <br>\n &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0x6A, 0x0D, 0x27, 0x3B, 0xEE&gt;&gt;)<br>\n\"16UwLL9Risc3QfPqBUvKofHmBQ7wMtjvM\"</code></pre>\n<p>Perfect!</p>\n<h2>Wrapping Up</h2>\n<p>If you’d like to see both modules in their full glory, I’ve included them in my <a href=\"https://github.com/pcorey/hello_bitcoin\"><code>hello_bitcoin</code> repository on Github</a>. Here’s a direct link to the <a href=\"https://github.com/pcorey/hello_bitcoin/blob/master/lib/base58.ex\"><code>Base58</code> module</a>, and the <a href=\"https://github.com/pcorey/hello_bitcoin/blob/master/lib/base58check.ex\"><code>Base58Check</code> module</a>, along with a <a href=\"https://github.com/pcorey/hello_bitcoin/blob/master/test/base58check_test.exs\">simple unit test</a>. If that repository looks familiar, it’s because it was used in a previous article on <a href=\"http://www.petecorey.com/blog/2017/09/04/controlling-a-bitcoin-node-with-elixir/\">controlling a Bitcoin node with Elixir</a>.</p>\n<p>I highly suggest you read through <a href=\"http://amzn.to/2E6gO0I\">Andreas Antonopoulos’ Mastering Bitcoin book</a> if you’re at all interested in how the Bitcoin blockchain works, or Bitcoin development in general. His book has been my primary source of inspiration and information for every Bitcoin article I’ve written to date.</p>\n<p><em>Originally published at</em> <a href=\"http://www.petecorey.com/blog/2018/01/08/bitcoins-base58check-in-pure-elixir/\"><em>www.petecorey.com</em></a> <em>on January 8, 2018.</em></p>\n</html>",
      "json_metadata": "{\"tags\":[\"programming\",\"bitcoin\"],\"links\":[\"http://amzn.to/2E6gO0I\",\"https://en.bitcoin.it/wiki/Base58Check_encoding\",\"https://en.bitcoin.it/wiki/Base58Check_encoding#Base58_symbol_chart\",\"https://hexdocs.pm/elixir/Base.html#encode64/2\",\"https://en.wikipedia.org/wiki/Base64#Examples\",\"http://erlang.org/doc/man/binary.html#decode_unsigned-1\",\"http://erlang.org/doc/man/erlang.html#binary_to_list-1\",\"https://hexdocs.pm/elixir/Enum.html#find_index/2\",\"http://erlang.org/doc/man/crypto.html#hash-2\",\"https://en.wikipedia.org/wiki/SHA-2\",\"https://en.bitcoin.it/wiki/Technical_background_of_version_1_Bitcoin_addresses\",\"https://github.com/pcorey/hello_bitcoin\",\"https://github.com/pcorey/hello_bitcoin/blob/master/lib/base58.ex\",\"https://github.com/pcorey/hello_bitcoin/blob/master/lib/base58check.ex\",\"https://github.com/pcorey/hello_bitcoin/blob/master/test/base58check_test.exs\",\"http://www.petecorey.com/blog/2017/09/04/controlling-a-bitcoin-node-with-elixir/\",\"http://www.petecorey.com/blog/2018/01/08/bitcoins-base58check-in-pure-elixir/\"],\"app\":\"steemit/0.1\",\"format\":\"html\"}"
    }
  ]
}

{
  "trx_id": "923682aeb7bff01782e84883728bdba575b2786b",
  "block": 19229554,
  "trx_in_block": 1,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2018-01-23T12:30:36",
  "op": [
    "transfer",
    {
      "from": "money-dreamer",
      "to": "petecorey",
      "amount": "0.001 STEEM",
      "memo": "Gift!"
    }
  ]
}

{
  "trx_id": "5017ed847aaab602cd3b6504b0f758af0e444294",
  "block": 19047122,
  "trx_in_block": 11,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2018-01-17T04:24:54",
  "op": [
    "comment",
    {
      "parent_author": "petecorey",
      "parent_permlink": "exploring-the-bitcoin-blockchain-with-elixir-and-phoenix",
      "author": "rodmclaughlin",
      "permlink": "re-petecorey-exploring-the-bitcoin-blockchain-with-elixir-and-phoenix-20180117t034438275z",
      "title": "",
      "body": "@@ -3338,8 +3338,175 @@\n -elixir/\n+%0A%0AEDIT: 'getinfo' has also been deprecated. It's now %0Acurl --data-binary '%7B%22jsonrpc%22:%221.0%22,%22method%22:%22getblockchaininfo%22,%22params%22:%5B%5D%7D'  http://user:pass@localhost:8332/\n",
      "json_metadata": "{\"tags\":[\"bitcoin\"],\"links\":[\"http://www.petecorey.com/blog/2017/09/04/controlling-a-bitcoin-node-with-elixir\",\"https://github.com/pcorey/hello_blockchain\",\"http://www.petecorey.com/blog/2017/09/04/controlling-a-bitcoin-node-with-elixir/\",\"http://user\"],\"app\":\"steemit/0.1\"}"
    }
  ]
}

{
  "trx_id": "ca8c234835caab57ea9c847392491e6f3645e5a8",
  "block": 19046597,
  "trx_in_block": 39,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2018-01-17T03:58:39",
  "op": [
    "comment",
    {
      "parent_author": "petecorey",
      "parent_permlink": "exploring-the-bitcoin-blockchain-with-elixir-and-phoenix",
      "author": "rodmclaughlin",
      "permlink": "re-petecorey-exploring-the-bitcoin-blockchain-with-elixir-and-phoenix-20180117t034438275z",
      "title": "",
      "body": "@@ -3140,12 +3140,206 @@\n r attention.\n+%0A%0AEDIT - just one more point. bitcoind now says rpcuser and rpcpassword will soon be deprecated, which you use in http://www.petecorey.com/blog/2017/09/04/controlling-a-bitcoin-node-with-elixir/\n",
      "json_metadata": "{\"tags\":[\"bitcoin\"],\"links\":[\"http://www.petecorey.com/blog/2017/09/04/controlling-a-bitcoin-node-with-elixir\",\"https://github.com/pcorey/hello_blockchain\",\"http://www.petecorey.com/blog/2017/09/04/controlling-a-bitcoin-node-with-elixir/\"],\"app\":\"steemit/0.1\"}"
    }
  ]
}

{
  "trx_id": "c6eb30f5f7302c01eb5b7db3390289f8a88477cb",
  "block": 19046317,
  "trx_in_block": 20,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2018-01-17T03:44:39",
  "op": [
    "comment",
    {
      "parent_author": "petecorey",
      "parent_permlink": "exploring-the-bitcoin-blockchain-with-elixir-and-phoenix",
      "author": "rodmclaughlin",
      "permlink": "re-petecorey-exploring-the-bitcoin-blockchain-with-elixir-and-phoenix-20180117t034438275z",
      "title": "",
      "body": "This is the best introduction I could find to both Phoenix and Blockchain. \n\nBut it assumes too much familiarity. There are places where it's not clear exactly what you have to do.\n\nAfter \n\"mix phx.new hello_blockchain --no-ecto\"\nit should say how to respond to the question \"Install dependencies Y/n?\"\n\nBefore \n\"mix phx.gen.html Blockchain Header headers --no-schema\"\nit should say\n\"cd hello_blockchain\"\n\nWhen it says\n\"Let’s remove the auto-generated contents of Blockchain and copy over the bitcoin_rpc function.\"\nit should say which file it is referring to. \n\n\"Be sure to add dependencies on :httpoison and :poison, and set up your :bitcoin_url in your configuration file.\"\nWhere do you add dependencies, and how?\nWhich file is your configuration file, how do you deduce what :bitcoin_url should be set up to, and how do you set it up?\n\n(Reading your post \"Controlling a Bitcoin Node with Elixir\", http://www.petecorey.com/blog/2017/09/04/controlling-a-bitcoin-node-with-elixir, makes it clear - but it would do no harm to repeat the information in the current post).\n\n\"let’s add their new routes to our router\"\nI know it's router.ex, but someone else might not.\n\n\"We’ll start with the BlockController\"\nI couldn't find it. Then I realised it's because I hadn't run\n\"mix phx.gen.html Blockchain Blocks blocks --no-schema\"\nbecause I'd missed it because it is not in a fixed width font like the other commands. \n\nOnce I'd run that command, I had \nlib/hello_blockchain_web/controllers/blocks_controller.ex\nand could follow the instructions about modifying \"BlockController\". Except, at this point, as a result of copying, pasting and running the commands above, I realised I had references to both BlockController and BlocksController in my project. I guessed it should be just \"BlocksController\", but I looked in https://github.com/pcorey/hello_blockchain, and found it was \"BlockController\", in files\ntest/hello_blockchain_web/controllers/block_controller_test.exs\nand\nlib/hello_blockchain_web/controllers/block_controller.ex\nIn fact there are no files beginning \"blocks\", and no classes beginning \"Blocks\".\n\nI think the section beginning \"Similarly, the index function in the HeaderController\" implies deleting all methods in \nlib/hello_blockchain_web/controllers/header_controller.ex\nand replacing them with the index and show methods described in that section, but if so, I think it should say so.\n\n\"There’s one final route we need to implement. When a user hits our application for the first time, they’ll land on the Phoenix landing page. Instead, let’s show them the most recent block header:\n\ndef index...\"\n\nAgain, it would help if it said which file this method should be implemented in.\n\nAnd \"<code><%= Poison.encode!(@block, pretty: true) %></code>\".\n\nAnd when it says change app.css, it would be helpful if it said whether that is \npriv/static/css/app.css\nor\nassets/css/app.css\nalthough, on second thoughts, it's obviously the latter.\n\nI hope you don't find this too pernickety. Most of it I figured out, but I did Rails for years. I suspect some readers would get it all straight away, and some would get lost.\n\nThanks for your attention.",
      "json_metadata": "{\"tags\":[\"bitcoin\"],\"links\":[\"http://www.petecorey.com/blog/2017/09/04/controlling-a-bitcoin-node-with-elixir\",\"https://github.com/pcorey/hello_blockchain\"],\"app\":\"steemit/0.1\"}"
    }
  ]
}

{
  "trx_id": "17c0c641a99ab4467c4f563a9044749ff9de7582",
  "block": 16299793,
  "trx_in_block": 1,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-10-13T17:20:36",
  "op": [
    "vote",
    {
      "voter": "richdevil",
      "author": "petecorey",
      "permlink": "ship-it-right-meow",
      "weight": 10000
    }
  ]
}

{
  "trx_id": "45f0c4c4f01a698ed16a82be9db02af7bac374e2",
  "block": 16299251,
  "trx_in_block": 5,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-10-13T16:53:27",
  "op": [
    "vote",
    {
      "voter": "jimy74",
      "author": "petecorey",
      "permlink": "ship-it-right-meow",
      "weight": 10000
    }
  ]
}

{
  "trx_id": "c9c6ecdb3468584bcb9c8fffe2c2dec99c60e315",
  "block": 16299203,
  "trx_in_block": 17,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-10-13T16:51:03",
  "op": [
    "comment",
    {
      "parent_author": "",
      "parent_permlink": "drawing",
      "author": "petecorey",
      "permlink": "ship-it-right-meow",
      "title": "Ship it right meow!",
      "body": "![shipit.png](https://steemitimages.com/DQmXuYvUqdPCw6LLCEiMBL4cJia7729DgPgW7fT2L7X3Cdk/shipit.png)",
      "json_metadata": "{\"tags\":[\"drawing\",\"programming\"],\"image\":[\"https://steemitimages.com/DQmXuYvUqdPCw6LLCEiMBL4cJia7729DgPgW7fT2L7X3Cdk/shipit.png\"],\"app\":\"steemit/0.1\",\"format\":\"markdown\"}"
    }
  ]
}

{
  "trx_id": "bc5808f985a8145a26d668aee5457d067261aefe",
  "block": 16297498,
  "trx_in_block": 23,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-10-13T15:25:27",
  "op": [
    "comment",
    {
      "parent_author": "petecorey",
      "parent_permlink": "learning-to-crawl-building-a-bare-bones-web-crawler-with-elixir",
      "author": "steemitboard",
      "permlink": "steemitboard-notify-petecorey-20171013t152529000z",
      "title": "",
      "body": "Congratulations @petecorey! You have completed some achievement on Steemit and have been rewarded with new badge(s) :\n\n[![](https://steemitimages.com/70x80/http://steemitboard.com/notifications/posts.png)](http://steemitboard.com/@petecorey) Award for the number of posts published\n\nClick on any badge to view your own Board of Honor on SteemitBoard.\nFor more information about SteemitBoard, click [here](https://steemit.com/@steemitboard)\n\nIf you no longer want to receive notifications, reply to this comment with the word `STOP`\n\n> By upvoting this notification, you can help all Steemit users. Learn how [here](https://steemit.com/steemitboard/@steemitboard/http-i-cubeupload-com-7ciqeo-png)!",
      "json_metadata": "{\"image\":[\"https://steemitboard.com/img/notifications.png\"]}"
    }
  ]
}

{
  "trx_id": "66954be97f4fb01e960d938580f31ea54c723275",
  "block": 16295019,
  "trx_in_block": 15,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-10-13T13:21:24",
  "op": [
    "vote",
    {
      "voter": "kumaranvpl",
      "author": "petecorey",
      "permlink": "learning-to-crawl-building-a-bare-bones-web-crawler-with-elixir",
      "weight": 10000
    }
  ]
}

{
  "trx_id": "45199561456f392d6ac33723db0d57d570e71aac",
  "block": 16294387,
  "trx_in_block": 1,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-10-13T12:49:48",
  "op": [
    "comment",
    {
      "parent_author": "petecorey",
      "parent_permlink": "learning-to-crawl-building-a-bare-bones-web-crawler-with-elixir",
      "author": "cheetah",
      "permlink": "cheetah-re-petecoreylearning-to-crawl-building-a-bare-bones-web-crawler-with-elixir",
      "title": "",
      "body": "Hi! I am a robot. I just upvoted you! I found similar content that readers might be interested in:\nhttp://www.east5th.co/blog/2017/10/09/learning-to-crawl-building-a-bare-bones-web-crawler-with-elixir/",
      "json_metadata": ""
    }
  ]
}

{
  "trx_id": "724a6b30ba7e68a1e3c23b2b5572cc9dcf4e9b4c",
  "block": 16294385,
  "trx_in_block": 36,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-10-13T12:49:42",
  "op": [
    "vote",
    {
      "voter": "cheetah",
      "author": "petecorey",
      "permlink": "learning-to-crawl-building-a-bare-bones-web-crawler-with-elixir",
      "weight": 50
    }
  ]
}

{
  "trx_id": "788fb57cc4b491334ae908bcedf1b2f717b878b7",
  "block": 16294379,
  "trx_in_block": 1,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-10-13T12:49:24",
  "op": [
    "comment",
    {
      "parent_author": "",
      "parent_permlink": "programming",
      "author": "petecorey",
      "permlink": "learning-to-crawl-building-a-bare-bones-web-crawler-with-elixir",
      "title": "Learning to Crawl - Building a Bare Bones Web Crawler with Elixir",
      "body": "# Learning to Crawl - Building a Bare Bones Web Crawler with Elixir\n&nbsp;\n> Originally posted on East5th. [Check out the original post.](http://www.east5th.co/blog/2017/10/09/learning-to-crawl-building-a-bare-bones-web-crawler-with-elixir/)\n\nI’ve been cooking up a side project recently that involves crawling through a domain, searching for links to specific websites.\n\nWhile I’m keeping the details of the project shrouded in mystery for now, building out a web crawler using Elixir sounds like a fantastic learning experience.\n\nLet’s roll up our sleeves and dig into it!\n\n## Let’s Think About What We Want\n&nbsp;\nBefore we start writing code, it’s important to think about what we want to build.\n\nWe’ll kick off the crawling process by handing our web crawler a starting URL. The crawler will fetch the page at that URL and look for links (`<a>` tags with `href` attributes) to other pages.\n\nIf the crawler finds a link to a page on the same domain, it’ll repeat the crawling process on that new page. This second crawl is considered one level “deeper” than the first. Our crawler should have a configurable maximum depth.\n\nOnce the crawler has traversed all pages on the given domain up to the specified depth, it will return a list of all internal and external URLs that the crawled pages link to.\n\nTo keep things efficient, let’s try to parallelize the crawling process as much as possible.\n\n## Hello, Crawler\n&nbsp;\nLet’s start off our crawler project by creating a new Elixir project:\n\n\tmix new hello_crawler\n&nbsp;\nWhile we’re at it, let’s pull in two dependencies we’ll be using in this project: [HTTPoison](https://hex.pm/packages/httpoison), a fantastic HTTP client for Elixir, and [Floki](https://hex.pm/packages/floki), a simple HTML parser.\n\nWe’ll add them to our `mix.exs` file:\n\n\tdefp deps do\n\t  [\n\t    {:httpoison, \"~> 0.13\"},\n\t    {:floki, \"~> 0.18.0\"}\n\t  ]\n\tend\n&nbsp;\nAnd pull them into our project by running a [Mix command](https://elixir-lang.org/getting-started/mix-otp/introduction-to-mix.html) from our shell:\n\n\tmix deps.get\n&nbsp;\nNow that we’ve set up the laid out the basic structure of our project, let’s start writing code!\n\n## Crawling a Page\n&nbsp;\nBased on [the description given above](#lets-think-about-what-we-want), we know that we’ll need some function (or set of functions) that takes in a starting URL, fetches the page, and looks for links.\n\nLet’s start sketching out that function:\n\n\tdef get_links(url) do\n\t  [url]\n\tend\n&nbsp;\nSo far our function just returns a list holding URL that was passed into it. That’s a start.\n\nWe’ll use [`HTTPoison`](https://hex.pm/packages/httpoison) to fetch the page at that URL (being sure to specify that we want to follow `301` redirects), and pull the resulting HTML out of the `body` field of the response:\n\n\twith {:ok, %{body: body}} <- HTTPoison.get(url, [], follow_redirect: true) do\n\t  [url]\n\telse\n\t  _ -> [url]\n\tend\n&nbsp;\nNotice that we’re using a `with` block and pattern matching on a successful call to `HTTPoison.get`. If a failure happens anywhere in our process, we abandon ship and return the current `url` in a list.\n\nNow we can use [`Floki`](https://hex.pm/packages/floki) to search for any `<a>` tags within our HTML and grab their `href` attributes:\n\n\twith {:ok, %{body: body}} <- HTTPoison.get(url, [], [follow_redirect: true]),\n\t     tags                 <- Floki.find(body, \"a\"),\n\t     hrefs                <- Floki.attribute(tags, \"href\") do\n\t  [url | hrefs]\n\telse\n\t  _ -> [url]\n\tend\n&nbsp;\nLastly, let’s clean up our code by replacing our `with` block with a new `handle_response` function with multiple function heads to handle success and failure:\n\n\tdef handle_response({:ok, %{body: body}}, url) do\n\t  [url | body\n\t         |> Floki.find(\"a\")\n\t         |> Floki.attribute(\"href\")]\n\tend\n\t\n\tdef handle_response(_response, url) do\n\t  [url]\n\tend\n\t\n\tdef get_links(url) do\n\t  headers = []\n\t  options = [follow_redirect: true]\n\t  url\n\t  |> HTTPoison.get(headers, options)\n\t  |> handle_response(url)\n\tend\n&nbsp;\nThis step is purely a stylistic choice. I find `with` blocks helpful for sketching out solutions, but prefer the readability of branching through pattern matching.\n\n![hello-crawler.png](https://steemitimages.com/DQmPrky6B7LBiR1hBhNWDDoL2CbJJLmr42VPb6WYDRNHfsH/hello-crawler.png)\n\nRunning our `get_links` function against a familiar site should return a handful of internal and external links:\n\n\tiex(1)> HelloCrawler.get_links(\"http://www.east5th.co/\")\n\t[\"http://www.east5th.co/\", \"/\", \"/blog/\", \"/our-work/\", \"/services/\", \"/blog/\",\n\t \"/our-work/\", \"/services/\",\n\t \"https://www.google.com/maps/place/Chattanooga,+TN/\", \"http://www.amazon.com/\",\n\t \"https://www.cigital.com/\", \"http://www.tapestrysolutions.com/\",\n\t \"http://www.surgeforward.com/\", \"/blog/\", \"/our-work/\", \"/services/\"]\n&nbsp;\nQuick, someone take a picture; it’s crawling!\n\n## Crawling Deeper\n&nbsp;\nWhile it’s great that we can crawl a single page, we need to go deeper. We want to scape links from our starting page and recursively scrape any other pages we’re linked to.\n\nThe most naive way of accomplishing this would be to recurse on `get_links` for every new list of links we find:\n\n\t[url | body\n\t       |> Floki.find(\"a\")\n\t       |> Floki.attribute(\"href\")\n\t       |> Enum.map(&get_links/1) # Recursive call get_links\n\t       |> List.flatten]\n&nbsp;\nWhile this works conceptually, it has a few problems:\n\n- Relative URLs, like `/services`, don’t resolve correctly in our call to `HTTPoison.get`.\n- A page that links to itself, or a set of pages that form a loop, will cause our crawler to enter an infinite loop.\n- We’re not restricting crawling to our original host.\n- We’re not counting depth, or enforcing any depth limit.\n\nLet’s address each of these issues.\n\n## Handling Relative URLs\n&nbsp;\nMany links within a page are relative; they don’t specify all of the necessary information to form a full URL.\n\nFor example, on `http://www.east5th.co/`, there’s a link to `/blog/`. Passing `\"/blog/\"` into `HTTPoison.get` will return an error, as `HTTPoison` doesn’t know the context for this relative address.\n\nWe need some way of transforming these relative links into full-blown URLs given the context of the page we’re currently crawling. Thankfully, Elixir’s standard library ships with [the fantastic `URI` module](https://hexdocs.pm/elixir/URI.html) that can help us do just that!\n\nLet’s use `URI.merge` and `to_string` to transform all of the links scraped by our crawler into well-formed URLs that can be understood by `HTTPoison.get`:\n\n\tdef handle_response({:ok, %{body: body}}, url) do\n\t  [url | body\n\t         |> Floki.find(\"a\")\n\t         |> Floki.attribute(\"href\")\n\t         |> Enum.map(&URI.merge(url, &1)) # Merge our URLs\n\t         |> Enum.map(&to_string/1)        # Convert the merged URL to a string\n\t         |> Enum.map(&get_links/1)\n\t         |> List.flatten]\n\tend\n&nbsp;\nNow our link to `/blog/` is transformed into `http://www.east5th.co/blog/` before being passed into `get_links` and subsequently `HTTPoison.get`.\n\nPerfect.\n\n## Preventing Loops\n&nbsp;\nWhile our crawler is correctly resolving relative links, this leads directly to our next problem: our crawler can get trapped in loops.\n\nThe first link on `http://www.east5th.co/` is to `/`. This relative link is translated into `http://www.east5th.co/`, which is once again passed into `get_links`, causing an infinite loop.\n\nTo prevent looping in our crawler, we’ll need to maintain a list of all of the pages we’ve already crawled. Before we recursively crawl a new link, we need to verify that it hasn’t been crawled previously.\n\nWe’ll start by adding a new, private, function head for `get_links` that accepts a `path` argument. The `path` argument holds all of the URLs we’ve visited on our way to the current URL.\n\n\tdefp get_links(url, path) do\n\t  headers = []\n\t  options = [follow_redirect: true]\n\t  url\n\t  |> HTTPoison.get(headers, options)\n\t  |> handle_response(url, path)\n\tend\n&nbsp;\nWe’ll call our new private `get_links` function from our public function head:\n\n\tdef get_links(url) do\n\t  get_links(url, []) # path starts out empty\n\tend\n&nbsp;\nNotice that our `path` starts out as an empty list.\n\n---- \n\nWhile we’re refactoring `get_links`, let’s take a detour and add a third argument called `context`:\n\n\tdefp get_links(url, path, context) do\n\t  url\n\t  |> HTTPoison.get(context.headers, context.options)\n\t  |> handle_response(url, path, context)\n\tend\n&nbsp;\nThe `context` argument is a map that lets us cleanly pass around configuration values without having to drastically increase the size of our function signatures. In this case, we’re passing in the `headers` and `options` used by `HTTPoison.get`.\n\nTo populate `context`, let’s change our public `get_links` function to build up the map by merging user-provided options with defaults provided by the module:\n\n\tdef get_links(url, opts \\\\ []) do\n\t  context = %{\n\t    headers: Keyword.get(opts, :headers, @default_headers),\n\t    options: Keyword.get(opts, :options, @default_options)\n\t  }\n\t  get_links(url, [], context)\n\tend\n&nbsp;\nIf the user doesn’t provide a value for `headers`, or `options`, we’ll use the defaults specified in the `@default_headers` and `@default_options` module attributes:\n\n\t@default_headers []\n\t@default_options [follow_redirect: true]\n&nbsp;\nThis quick refactor will help keep things clean going down the road.\n\n---- \n\nWhenever we crawl a URL, we’ll add that URL to our `path`, like a breadcrumb marking where we’ve been.\n\n![crawler-path.png](https://steemitimages.com/DQmYGPUmFP23Bk1ZeTK1tnWDtErcDpKb1TL8qaG1HRR5HzG/crawler-path.png)\n\nNext, we’ll filter out any links we find that we’ve already visited, and append the current `url` to `path` before recursing on `get_links`:\n\n\tpath = [url | path] # Add a breadcrumb\n\t[url | body\n\t       |> Floki.find(\"a\")\n\t       |> Floki.attribute(\"href\")\n\t       |> Enum.map(&URI.merge(url, &1))\n\t       |> Enum.map(&to_string/1)\n\t       |> Enum.reject(&Enum.member?(path, &1)) # Avoid loops\n\t       |> Enum.map(&get_links(&1, [&1 | path], context))\n\t       |> List.flatten]\n&nbsp;\nIt’s important to note that this doesn’t completely prevent the repeated fetching of the same page. It simply prevents the same page being refetched in a single recursive chain, or path.\n\nImagine if page A links to pages B and C. If page B or C link back to page A, page A will not be refetched. However, if both page B and page C link to page D, page D will be fetched twice.\n\nWe won’t worry about this problem in this article, but caching our calls to `HTTPoison.get` might be a good solution.\n\n## Restricting Crawling to a Host\n&nbsp;\nIf we try and run our new and improved `WebCrawler.get_links` function, we’ll notice that it takes a long time to run. __In fact in most cases, it’ll never return!__\n\nThe problem is that we’re not limiting our crawler to crawl only pages within our starting point’s domain. If we crawl `http://www.east5th.co/`, we’ll eventually get linked to Google and Amazon, and from there, the crawling never ends.\n\nWe need to detect the host of the starting page we’re given, and restrict crawling to only that host.\n\nThankful, the `URI` module once again comes to the rescue.\n\nWe can use `URI.parse` to pull out the `host` of our starting URL, and pass it into each call to `get_links` and `handle_response` via our `context` map:\n\n\tdef get_links(url, opts \\\\ []) do\n\t  url = URI.parse(url)\n\t  context = %{\n\t    ...\n\t    host: url.host # Add our initial host to our context\n\t  }\n\t  get_links(url, [], context)\n\tend\n&nbsp;\nUsing the parsed `host`, we can check if the `url` passed into `get_links` is a crawlable URL:\n\n\tdefp get_links(url, path, context) do\n\t  if crawlable_url?(url, context) do # check before we crawl\n\t    ...\n\t  else\n\t    [url]\n\t  end\n\tend\n&nbsp;\nThe `crawlable_url?` function simply verifies that the host of the URL we’re attempting to crawl matches the host of our initial URL, passed in through our `context`:\n\n\tdefp crawlable_url?(%{host: host}, %{host: initial}) when host == initial, do: true\n\tdefp crawlable_url?(_, _), do: false\n&nbsp;\nThis guard prevents us from crawling external URLs.\n\n---- \n\nBecause we passed in the result of `URI.parse` into `get_links`, our `url` is now a `URI` struct, rather than a string. We need to convert it back into a string before passing it into `HTTPoison.get` and `handle_response`:\n\n\tif crawlable_url?(url, context) do\n\t  url\n\t  |> to_string # convert our %URI to a string\n\t  |> HTTPoison.get(context.headers, context.options)\n\t  |> handle_response(path, url, context)\n\telse\n\t  [url]\n\tend\n&nbsp;\nAdditionally, you’ve probably noticed that our collected list of links is no longer a list of URL strings. Instead, it’s a list of `URI` structs.\n\nAfter we finish crawling through our target site, let’s convert all of the resulting structs back into strings, and remove any duplicates:\n\n\tdef get_links(url, opts \\\\ []) do\n\t  ...\n\t  get_links(url, [], context)\n\t  |> Enum.map(&to_string/1) # convert URI structs to strings\n\t  |> Enum.uniq # remove any duplicate urls\n\tend\n&nbsp;\nWe’re crawling deep now!\n\n## Limiting Our Crawl Depth\n&nbsp;\nOnce again, if we let our crawler loose on the world, it’ll most likely take a long time to get back to us.\n\nBecause we’re not limiting how deep our crawler can traverse into our site, it’ll explore all possible paths [that don’t involve retracing its steps](#preventing-loops). We need a way of telling it not to continue crawling once it’s already followed a certain number of links and reached a specified depth.\n\nDue to the way we structured out solution, __this is incredibly easy to implement!__\n\nAll we need to do is add another guard in our `get_links` function that makes sure that the length of `path` hasn’t exceeded our desired depth:\n\n\tif continue_crawl?(path, context) and crawlable_url?(url, context) do\n&nbsp;\nThe `continue_crawl?` function verifies that the length of `path` hasn’t grown past the `max_depth` value in our `context` map:\n\n\tdefp continue_crawl?(path, %{max_depth: max}) when length(path) > max, do: false\n\tdefp continue_crawl?(_, _), do: true\n&nbsp;\nIn our public `get_links` function we can add `max_depth` to our `context` map, pulling the value from the user-provided `opts` or falling back to the `@default_max_depth` module attribute:\n\n\t@default_max_depth 3\n&nbsp;\n\n\tcontext = %{\n\t  ...\n\t  max_depth: Keyword.get(opts, :max_depth, @default_max_depth)\n\t}\n&nbsp;\nAs with our other `opts`, the default `max_depth` can be overridden by passing a custom `max_depth` value into `get_links`:\n\n\tHelloCrawler.get_links(\"http://www.east5th.co/\", max_depth: 5)\n&nbsp;\nIf our crawler tries to crawl deeper than the maximum allowed depth, `continue_crawl?` returns `false` and `get_links` returns the `url` being crawled, preventing further recursion.\n\nSimple and effective.\n\n## Crawling in Parallel\n&nbsp;\nWhile our crawler is fully functional at this point, it would be nice to improve upon it a bit. Instead of waiting for every path to be exhausted before crawling the next path, why can’t we crawl multiple paths in parallel?\n\nAmazingly, parallelizing our web crawler is as simple as swapping our  map over the links we find on a page with a [parallelized map](http://elixir-recipes.github.io/concurrency/parallel-map/):\n\n\t|> Enum.map(&(Task.async(fn -> get_links(URI.parse(&1), [&1 | path], context) end)))\n\t|> Enum.map(&Task.await/1)\n&nbsp;\nInstead of passing each scraped link into `get_links` and waiting for it to fully crawl every sub-path before moving onto the next link, we pass all of our links into asynchronous calls to `get_links`, and then wait for all of those asynchronous calls to return.\n\nFor every batch of crawling we do, we really only need to wait for the slowest page to be crawled, rather than waiting one by one for every page to be crawled.\n\nEfficiency!\n\n## Final Thoughts\n&nbsp;\nIt’s been a process, but we’ve managed to get our simple web crawler up and running.\n\nWhile we accomplished everything we set out to do, our final result is still very much a bare bones web crawler. A more mature crawler would come equipped with more sophisticated scraping logic, rate limiting, caching, and more efficient optimizations.\n\nThat being said, I’m proud of what we’ve accomplished. Be sure to check out [the entire `HelloCrawler` project on Github](https://github.com/pcorey/hello_crawler/blob/master/lib/hello_crawler.ex).\n\nI’d like to thank [Mischov](https://github.com/mischov) for his incredibly helpful suggestions for improving [this article](https://github.com/pcorey/pcorey.github.io/issues/31) and [the underlying codebase](https://github.com/pcorey/hello_crawler/issues/1). If you have any other suggestions, let me know!",
      "json_metadata": "{\"tags\":[\"programming\",\"elixir\",\"web\",\"crawling\"],\"image\":[\"https://steemitimages.com/DQmPrky6B7LBiR1hBhNWDDoL2CbJJLmr42VPb6WYDRNHfsH/hello-crawler.png\",\"https://steemitimages.com/DQmYGPUmFP23Bk1ZeTK1tnWDtErcDpKb1TL8qaG1HRR5HzG/crawler-path.png\"],\"links\":[\"http://www.east5th.co/blog/2017/10/09/learning-to-crawl-building-a-bare-bones-web-crawler-with-elixir/\",\"https://hex.pm/packages/httpoison\",\"https://hex.pm/packages/floki\",\"https://elixir-lang.org/getting-started/mix-otp/introduction-to-mix.html\",\"#lets-think-about-what-we-want\",\"https://hexdocs.pm/elixir/URI.html\",\"#preventing-loops\",\"http://elixir-recipes.github.io/concurrency/parallel-map/\",\"https://github.com/pcorey/hello_crawler/blob/master/lib/hello_crawler.ex\",\"https://github.com/mischov\",\"https://github.com/pcorey/pcorey.github.io/issues/31\",\"https://github.com/pcorey/hello_crawler/issues/1\"],\"app\":\"steemit/0.1\",\"format\":\"markdown\"}"
    }
  ]
}

{
  "trx_id": "44caedfa6709ad890a587abe62d1df0c0e80fb17",
  "block": 15864195,
  "trx_in_block": 24,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-09-28T14:05:33",
  "op": [
    "vote",
    {
      "voter": "ubg",
      "author": "petecorey",
      "permlink": "exploring-the-bitcoin-blockchain-with-elixir-and-phoenix",
      "weight": 100
    }
  ]
}

{
  "trx_id": "5f526bfa395c9823ffb4ee706d5f7acdb1932e6a",
  "block": 15863652,
  "trx_in_block": 11,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-09-28T13:38:24",
  "op": [
    "comment",
    {
      "parent_author": "petecorey",
      "parent_permlink": "exploring-the-bitcoin-blockchain-with-elixir-and-phoenix",
      "author": "forexmaster",
      "permlink": "re-petecorey-exploring-the-bitcoin-blockchain-with-elixir-and-phoenix-20170928t133830403z",
      "title": "",
      "body": "Great post. I'm Following. Please follow me...",
      "json_metadata": "{\"tags\":[\"bitcoin\"],\"app\":\"steemit/0.1\"}"
    }
  ]
}

{
  "trx_id": "0ec753f688168106b511e2ab1eef04bdcb0c3f51",
  "block": 15863641,
  "trx_in_block": 32,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-09-28T13:37:51",
  "op": [
    "comment",
    {
      "parent_author": "petecorey",
      "parent_permlink": "exploring-the-bitcoin-blockchain-with-elixir-and-phoenix",
      "author": "cheetah",
      "permlink": "cheetah-re-petecoreyexploring-the-bitcoin-blockchain-with-elixir-and-phoenix",
      "title": "",
      "body": "Hi! I am a robot. I just upvoted you! I found similar content that readers might be interested in:\nhttp://www.east5th.co/blog/2017/09/18/exploring-the-bitcoin-blockchain-with-elixir-and-phoenix/",
      "json_metadata": ""
    }
  ]
}

{
  "trx_id": "2dd9a50e19280d8ddd9cd13d0a5c042ff062cf4a",
  "block": 15863640,
  "trx_in_block": 9,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-09-28T13:37:48",
  "op": [
    "vote",
    {
      "voter": "cheetah",
      "author": "petecorey",
      "permlink": "exploring-the-bitcoin-blockchain-with-elixir-and-phoenix",
      "weight": 50
    }
  ]
}

{
  "trx_id": "dacd2df5bf42d261bb4fbc5b4348c637745af171",
  "block": 15863632,
  "trx_in_block": 3,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-09-28T13:37:24",
  "op": [
    "vote",
    {
      "voter": "petecorey",
      "author": "petecorey",
      "permlink": "exploring-the-bitcoin-blockchain-with-elixir-and-phoenix",
      "weight": 10000
    }
  ]
}

{
  "trx_id": "81d03a43008e217d5ecf49ae41d50035eb88cbb1",
  "block": 15863629,
  "trx_in_block": 19,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-09-28T13:37:15",
  "op": [
    "comment",
    {
      "parent_author": "",
      "parent_permlink": "bitcoin",
      "author": "petecorey",
      "permlink": "exploring-the-bitcoin-blockchain-with-elixir-and-phoenix",
      "title": "Exploring the Bitcoin Blockchain with Elixir and Phoenix",
      "body": "# Exploring the Bitcoin Blockchain with Elixir and Phoenix\n&nbsp;\nEarlier this month [we dove into the brave new world of Bitcoin development](http://www.east5th.co/blog/2017/09/04/controlling-a-bitcoin-node-with-elixir/) by writing [an Elixir module](https://github.com/pcorey/hello_bitcoin/blob/master/lib/hello_bitcoin.ex) that could communicate with a [Bitcoin full node](https://bitcoin.org/en/full-node). At this end of the day, we had a small handful of Elixir functions that could retrieve some basic information about the Bitcoin blockchain.\n\nLet’s expand on that idea a bit.\n\nIn this article we’re going to use the [Phoenix framework](http://phoenixframework.org/) to build a bare-bones blockchain viewer. Let’s get to it!\n\n## Project Scaffolding\n&nbsp;\nFirst things first, let’s create our new Phoenix project. We’ll be using [Phoenix 1.3](http://phoenixframework.org/blog/phoenix-1-3-0-released) and the new `phx` generators that shipped with it:\n\n\tmix phx.new hello_blockchain --no-ecto\n&nbsp;\nOnce we’ve gone ahead and set up our new project, let’s add two new routes to our application. One for viewing block headers, and another for viewing full blocks:\n\n\tmix phx.gen.html Blockchain Header headers --no-schema\n\t\n&nbsp;\n\tmix phx.gen.html Blockchain Blocks blocks --no-schema\n&nbsp;\nNotice that we specified `--no-ecto` when generating our new project, and `--no-schema` when generating our block and header resources. We don’t be needing [Ecto](https://github.com/elixir-ecto/ecto) in our blockchain viewer. All of the data we’re rendering lives in our full node!\n\n## Our Blockchain Context\n&nbsp;\nWhen we generated our header and block resources, we also generated a `Blockchain` context module. Our context will be our interfacing with our Bitcoin full node.\n\nSound familiar?\n&nbsp;\nThat’s because we implemented the `Blockchain` context module in our last article! Let’s remove the auto-generated contents of `Blockchain` and copy over [the `bitcoin_rpc` function](https://github.com/pcorey/hello_bitcoin/blob/master/lib/hello_bitcoin.ex#L3-L15).\n\n\tdef bitcoin_rpc(method, params \\\\ []) do\n\t  with url <- Application.get_env(:hello_bitcoin, :bitcoin_url),\n\t       command <- %{jsonrpc: \"1.0\", method: method, params: params},\n\t       {:ok, body} <- Poison.encode(command),\n\t       {:ok, response} <- HTTPoison.post(url, body),\n\t       {:ok, metadata} <- Poison.decode(response.body),\n\t       %{\"error\" => nil, \"result\" => result} <- metadata do\n\t    {:ok, result}\n\t  else\n\t    %{\"error\" => reason} -> {:error, reason}\n\t    error -> error\n\t  end\n\tend\n&nbsp;\nBe sure to add dependencies on `:httpoison` and `:poison`, and set up your `:bitcoin_url` in your configuration file.\n\nOnce we’ve finished that, we’ll add four helper functions to our `Blockchain` module. we’ll use these functions to fetch the data we want to render with our blockchain viewer:\n\n\tdef getbestblockhash, do: bitcoin_rpc(\"getbestblockhash\")\n\t\n\tdef getblockhash(height), do: bitcoin_rpc(\"getblockhash\", [height])\n\t\n\tdef getblock(hash), do: bitcoin_rpc(\"getblock\", [hash])\n\t\n\tdef getblockheader(hash), do: bitcoin_rpc(\"getblockheader\", [hash])\n&nbsp;\nNow that we have a functional context module, we can start wiring it up to our routes.\n\n## Routing\n&nbsp;\nArmed with our newly generated block and header resources, let’s add their new routes to our router:\n\n\tscope \"/\", HelloBlockchainWeb do\n\t  pipe_through :browser\n\t  resources \"/\", PageController, only: [:index]\n\t  resources \"/blocks\", BlockController, only: [:index, :show]\n\t  resources \"/headers\", HeaderController, only: [:index, :show]\n\tend\n&nbsp;\nFor now, we’re only going to be handling the `:index` and `:show` routes for our blocks and headers.\n\nNow that our routes are established, let’s move on to refactoring our controller modules. We’ll start with the `BlockController`.\n\nWe’re going to remove all of our controller functions except the `index` and `show` functions, which we’ll heavily modify.\n\nWhen we hit the `/blocks/` route, we’ll fall into the `index` controller function. We want this function to redirect us to the most recent block in the blockchain:\n\n\tdef index(conn, _params) do\n\t  with {:ok, hash} <- Blockchain.getbestblockhash() do\n\t    redirect(conn, to: block_path(conn, :show, hash))\n\t  end\n\tend\n&nbsp;\nWe use `getbestblockhash` to get the hash of the most recently validated Bitcoin block, and we redirect the user to the `:show` route, providing the resulting hash.\n\nOur `show` controller function accepts the provided `hash` as an argument, fetches more information about the block from our full node, and finally renders the block using the `show.html` template:\n\n\tdef show(conn, %{\"id\" => hash}) do\n\t  with {:ok, block} <- Blockchain.getblock(hash) do\n\t    render(conn, \"show.html\", block: block)\n\t  end\n\tend\n\n&nbsp;\n\n---- \n&nbsp;\nSimilarly, the `index` function in the `HeaderController` redirects the user to the `:show` route of the most recently verified block:\n\n\tdef index(conn, _params) do\n\t  with {:ok, hash} <- Blockchain.getbestblockhash() do\n\t    redirect(conn, to: header_path(conn, :show, hash))\n\t  end\n\tend\n&nbsp;\nWhile the `show` function fetches the relevant information using `Blockchain.getblockheader` and passes it into its `show.html` template:\n\n\tdef show(conn, %{\"id\" => hash}) do\n\t  with {:ok, block} <- Blockchain.getblockheader(hash) do\n\t    render(conn, \"show.html\", block: block)\n\t  end\n\tend\n&nbsp;\n\n---- \n\n&nbsp;\nThere’s one final route we need to implement. When a user hits our application for the first time, they’ll land on the Phoenix landing page. Instead, let’s show them the most recent block header:\n\n\tdef index(conn, _params) do\n\t  with {:ok, hash} <- Blockchain.getbestblockhash() do\n\t    redirect(conn, to: header_path(conn, :show, hash))\n\t  end\n\tend\n&nbsp;\nOnce again, we use `Blockchain.getbestblockhash` to fetch the most recently verified block hash from our Bitcoin full node. We use that hash to redirect the user to the header’s `:show` route.\n\nWith our routes properly configured, we can fetch data about any full block or block header just by knowings its hash.\n\nLet’s more on to the final piece of the puzzle: rendering that data.\n\n## Block and Header Templates\n&nbsp;\nOur blockchain viewer now correctly routes the user to the appropriate `:show` route for either the block or block header they’re trying to view, and passes all relevant data to the corresponding template to render.\n\nNow all we need to do is build out our templates!\n\nTo keep the scope of this article manageable, we’ll keep our user interface as minimal as possible. The more bare-bones way to render our blocks and headers, while still being meaningful to our users, is to render the data received from our controllers as JSON code blocks.\n\nThe simplest way to do this would be to dump the output of `Poison.encode!` into the DOM:\n\n\t<code><%= Poison.encode!(@block, pretty: true) %></code>\n&nbsp;\nThe `pretty: true` option passed into `Poison.encode!` ensures that the resulting JSON string is nicely formatted. In our `app.css` file, we should set the `white-space` rule on `<code>` blocks to preserve this formatting:\n\n\tcode {\n\t  white-space: pre !important;\n\t}\n&nbsp;\nBeautiful.\n\nA bit minimal, but beautiful none-the-less.\n&nbsp;\n\n---- \n\n&nbsp;\nWhile dumping raw JSON into the DOM is informative, it’s not especially user-friendly. Let’s add an extra layer of interactivity to our blockchain viewer.\n\nThe blocks and block headers we receive from our Bitcoin full node come with `previousblockhash` field and (usually) a `nextblockhash` field. These hashes, as we would expect, point to the previous and next blocks in the blockchain, respectively. Let’s transform these hashes into links so users can easily navigate through the blockchain.\n\nThe first thing we’ll do is write a function in the corresponding view file that converts hashes into links. In our `HeaderView` module, our `hash_link` function would look like this:\n\n\tdefp hash_link(hash), do: \"<a href='/headers/#{hash}'>#{hash}</a>\"\n&nbsp;\nUsing this function, we can write a function that modifies our block header. It replaces the hashes in `previousblockhash` and `nextblockhash` with links to those blocks, and JSON encodes the resulting object:\n\n\tdef mark_up_block(block) do\n\t  block\n\t  |> Map.replace(\"previousblockhash\", hash_link(block[\"previousblockhash\"]))\n\t  |> Map.replace(\"nextblockhash\", hash_link(block[\"nextblockhash\"]))\n\t  |> Poison.encode!(pretty: true)\n\tend\n&nbsp;\nIn our HTML template, we can replace the contents of our `<code>` block with the result of `mark_up_block`:\n\n\t<code><%= raw(mark_up_block(@block)) %></code>\n&nbsp;\nNotice that we’re wrapping `mark_up_block` with `raw`. We want to HTML being injected into our JSON to be interpreted as raw HTML, and not encoded as special characters.\n\nAfter carrying out the same changes in our `BlockView` and our block HTMl template, cleaning up our layout template, and adding a few final styling touches, we have our result.\n\nBehold, the most basic of blockchain explorers!\n\n## Final Thoughts\n&nbsp;\nObviously, this is just the tip of the iceberg when it comes to building a Bitcoin blockchain explorer.\n\nI’m super excited about Bitcoin and development projects related to Bitcoin. If you’ve made it this far, I assume you are as well! If you’re looking for a deep dive into Bitcoin development, I recommend you check out the fantastic [Mastering Bitcoin](https://www.amazon.com/gp/product/1449374042/ref=as_li_qf_sp_asin_il_tl?ie=UTF8&tag=east5th-20&camp=1789&creative=9325&linkCode=as2&creativeASIN=1449374042&linkId=23b0becd76b444f25a75a3143f5d253f) (affiliate link) book by Andreas Antonopoulos.\n\nExpect more updates to this blockchain explorer in the future, and more Bitcoin focused projects. In the meantime, be sure to [check out this blockchain viewer project on Github](https://github.com/pcorey/hello_blockchain).",
      "json_metadata": "{\"tags\":[\"bitcoin\",\"programming\",\"cryptocurrency\"],\"links\":[\"http://www.east5th.co/blog/2017/09/04/controlling-a-bitcoin-node-with-elixir/\",\"https://github.com/pcorey/hello_bitcoin/blob/master/lib/hello_bitcoin.ex\",\"https://bitcoin.org/en/full-node\",\"http://phoenixframework.org/\",\"http://phoenixframework.org/blog/phoenix-1-3-0-released\",\"https://github.com/elixir-ecto/ecto\",\"https://github.com/pcorey/hello_bitcoin/blob/master/lib/hello_bitcoin.ex#L3-L15\",\"https://www.amazon.com/gp/product/1449374042/ref=as_li_qf_sp_asin_il_tl?ie=UTF8&tag=east5th-20&camp=1789&creative=9325&linkCode=as2&creativeASIN=1449374042&linkId=23b0becd76b444f25a75a3143f5d253f\",\"https://github.com/pcorey/hello_blockchain\"],\"app\":\"steemit/0.1\",\"format\":\"markdown\"}"
    }
  ]
}

{
  "trx_id": "020f5eb05e4e48181214d6f168476b953501b3cc",
  "block": 15235214,
  "trx_in_block": 29,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-09-06T17:43:36",
  "op": [
    "vote",
    {
      "voter": "petecorey",
      "author": "petecorey",
      "permlink": "controlling-a-bitcoin-node-with-elixir",
      "weight": 10000
    }
  ]
}

{
  "trx_id": "e62a47d3b8aec74da9235ef3561d75bebc1868ab",
  "block": 15234039,
  "trx_in_block": 1,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-09-06T16:44:51",
  "op": [
    "vote",
    {
      "voter": "kromtar",
      "author": "petecorey",
      "permlink": "controlling-a-bitcoin-node-with-elixir",
      "weight": 500
    }
  ]
}

{
  "trx_id": "f60b10384da40c6eebd592adf3e58fcd35c92501",
  "block": 15233434,
  "trx_in_block": 25,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-09-06T16:14:36",
  "op": [
    "comment",
    {
      "parent_author": "",
      "parent_permlink": "bitcoin",
      "author": "petecorey",
      "permlink": "controlling-a-bitcoin-node-with-elixir",
      "title": "Controlling a Bitcoin Node with Elixir",
      "body": "# Controlling a Bitcoin Node with Elixir\n&nbsp;\nI’ve been bit by the [Bitcoin](https://bitcoin.org/en/) bug, and I’ve been bit hard. To satiate my thirst for knowledge, I’ve been reading the fantastic [Mastering Bitcoin](https://www.amazon.com/gp/product/1449374042/ref=as_li_qf_sp_asin_il_tl?ie=UTF8&tag=east5th-20&camp=1789&creative=9325&linkCode=as2&creativeASIN=1449374042&linkId=23b0becd76b444f25a75a3143f5d253f) (affiliate link) book by Andreas Antonopoulos, and diving into the brave new world of Bitcoin development.\n\nMastering Bitcoin does a fantastic job of outlining the technical underpinnings of Bitcoin, but I wanted to solidify my understanding with some hands-on experience.\n\nWriting a simple [Elixir](https://elixir-lang.org/) application to communicate and control a Bitcoin Core full node through its [JSON-RPC interface](https://en.bitcoin.it/wiki/API_reference_(JSON-RPC)) seems like a fantastic “hello world” exercise. Let’s get to it!\n\n## You’ll Need a Full Node\n&nbsp;\nThe first step in communicating with a Bitcoin Core full node is getting our hands on one. While publicly available nodes with wide open JSON-RPC interfaces are few and far between, it’s fairly simple to [run our own Bitcoin Core node locally](https://bitcoin.org/en/full-node).\n\nAssuming we’ve [installed the `bitcoind` daemon on our system](https://bitcoin.org/en/full-node#osx-daemon), we’ll need to configure it with a `bitcoin.config` file:\n\n\trpcuser=<username>\n\trpcpassword=<password>\n&nbsp;\nThe `<username>` and `<password>` values we define in our configuration will be used to authenticate ourselves when making requests to the Bitcoin node.\n\nOnce we’ve created our configuration file, we can spin up our full node:\n\n\tbitcoind -conf=<path to bitcoin.config> -daemon\n&nbsp;\nOnce started, our full node daemon will begin connecting to peer nodes, downloading, and verifying blocks from the blockchain.\n\nWe can verify that everything is working as expected:\n\n\tbitcoin-cli getinfo\n&nbsp;\nThis command should return some basic information about the node, including the node’s `\"version\"`, and the number of `\"blocks\"` it’s received and verified. It may take several days to download and verify the entire blockchain, but we can keep continue on with our project in the meantime.\n\n## The Bitcoin Node’s JSON-RPC\n&nbsp;\nOur Bitcoin full node implements a [JSON-based RPC API](https://en.bitcoin.it/wiki/API_reference_(JSON-RPC)) which can be used to retrieve information about the Bitcoin blockchain, and to interact with the node itself.\n\nInterestingly, the `bitcoin-cli` tool that we used to get information about the node [leverages this JSON-RPC API](https://github.com/bitcoin/bitcoin/blob/master/src/bitcoin-cli.cpp#L194-L285). You can fetch a list of all of the available RPC commands on the node by calling `bitcoin-cli help`, or by browsing through the [Bitcoin Wiki](https://en.bitcoin.it/wiki/Original_Bitcoin_client/API_calls_list).\n\nThe node’s JSON-RPC accepts incoming commands through an HTTP server, which means that we can manually craft these RPC commands and bypass the `bitcoin-cli` tool entirely.\n\nFor example, [we can run `getinfo` manually with `curl`](https://bitcoin.stackexchange.com/a/19839):\n\n\tcurl --data-binary '{\"jsonrpc\":\"1.0\",\"method\":\"getinfo\",\"params\":[]}'  \\\n\t     http://<user>:<pass>@localhost:8332/\n&nbsp;\nSimilarly, we can execute these commands from any programming environment with an HTTP client, like Elixir!\n\n## Setting Up Our Elixir Application\n&nbsp;\nNow that we have a strategy for communicating with our Bitcoin full node, let’s start building out our Elixir application.\n\nFirst, we’ll create a new Elixir project and update our `mix.exs` to add dependencies on [`poison`](https://hex.pm/packages/poison), which we’ll need to encode and decode JSON objects, and [`httpoison`](https://hex.pm/packages/httpoison), our go-to Elixir HTTP client.\n\n\tdefp deps do\n\t  [\n\t    {:httpoison, \"~> 0.13\"},\n\t    {:poison, \"~> 3.1\"}\n\t  ]\n\tend\n&nbsp;\nNow that we’ve laid out the scaffolding for our application, let’s turn our attention towards talking with our Bitcoin node.\n\nWe’ll start by gutting our `HelloBitcoin` module, and stubbing out a new `getinfo` function:\n\n\tdefmodule HelloBitcoin do\n\t\n\t  def getinfo do\n\t    raise \"TODO: Implement getinfo\"\n\t  end\n\t\n\tend\n&nbsp;\nTo keep things simple, we’ll interact with this module through `iex -S mix`. As a sanity check, let’s verify that everything is working correctly before moving on to the next section.\n\nCalling our `HelloBitcoin.getinfo` stub should raise a runtime exception:\n\n\tiex(1)> HelloBitcoin.getinfo\n\tHelloBitcoin.getinfo\n\t** (RuntimeError) TODO: Implement getinfo\n\t    (hello_bitcoin) lib/hello_bitcoin.ex:4: HelloBitcoin.getinfo/0\n&nbsp;\nPerfect. Progress through failure.\n\n## Constructing the GetInfo Command\n&nbsp;\nLet’s start to flesh out our `getinfo` function.\n\nTo recap, our goal is to send a `POST` HTTP request to our Bitcoin node’s HTTP server (usually listening on `http://localhost:8332`), passing in a JSON object that holds the command we’re trying to execute and any required parameters.\n\nIt turns out this is incredibly easy with `httpoison`:\n\n\tdef getinfo do\n\t  with url     <- Application.get_env(:hello_bitcoin, :bitcoin_url),\n\t       command <- %{jsonrpc: \"1.0\", method: \"getinfo\", params: []},\n\t       body    <- Poison.encode!(command),\n\t       headers <- [{\"Content-Type\", \"application/json\"}] do\n\t    HTTPoison.post!(url, body, headers)\n\t  end\n\tend\n&nbsp;\nWe start by pulling our `url` from the `bitcoin_url` key in our application’s configuration. This needs to be set in `config/config.exs` and should point to your local node:\n\n\tconfig :hello_bitcoin, bitcoin_url: \"http://<user>:<password>@localhost:8332\"\n&nbsp;\nNext, we build a map that represents our JSON-RPC `command`. In this case, our `method` is `\"getinfo\"`, which requires no `params`. Finally, we construct the `body` of our request by JSON encoding our `command` with `Poison.encode!`.\n\nCalling `HelloBitcoin.getinfo` should give us a successful `200` response from the Bitcoin node, along with the JSON encoded response to our `getinfo` command:\n\n\t%HTTPoison.Response{\n\t  body: \"{\\\"result\\\":{\\\"version\\\":140200,\\\"protocolversion\\\":70015,\\\"walletversion\\\":130000,\\\"balance\\\":0.00000000,\\\"blocks\\\":482864,\\\"timeoffset\\\":-1,\\\"connections\\\":8,\\\"proxy\\\":\\\"\\\",\\\"difficulty\\\":888171856257.3206,\\\"testnet\\\":false,\\\"keypoololdest\\\":1503512537,\\\"keypoolsize\\\":100,\\\"paytxfee\\\":0.00000000,\\\"relayfee\\\":0.00001000,\\\"errors\\\":\\\"\\\"},\\\"error\\\":null,\\\"id\\\":null}\\n\",\n\t  headers: [{\"Content-Type\", \"application/json\"}, {\"Date\", \"Thu, 31 Aug 2017 21:27:02 GMT\"}, {\"Content-Length\", \"328\"}],\n\t  request_url: \"http://localhost:8332\",\n\t  status_code: 200\n\t}\n&nbsp;\nBeautiful.\n\nLet’s decode the resulting JSON in `body` and return the result:\n\n\tHTTPoison.post!(url, body)\n\t|> Map.get(:body)\n\t|> Poison.decode!\n&nbsp;\nNow our call to `HelloBitcoin.getinfo` returns the result returned by `bitcoind` in a more usable format:\n\n\t%{\"error\" => nil, \"id\" => nil,\n\t  \"result\" => %{\"balance\" => 0.0, \"blocks\" => 483001, \"connections\" => 8,\n\t    \"difficulty\" => 888171856257.3206, \"errors\" => \"\",\n\t    \"keypoololdest\" => 1503512537, \"keypoolsize\" => 100, \"paytxfee\" => 0.0,\n\t    \"protocolversion\" => 70015, \"proxy\" => \"\", \"relayfee\" => 1.0e-5,\n\t    \"testnet\" => false, \"timeoffset\" => -1, \"version\" => 140200,\n\t    \"walletversion\" => 130000}}\n&nbsp;\nYou’ll notice that the `\"result\"`, the data we actually want, is wrapped in a map containing metadata about the request itself. This metadata includes a potential `error` string, and the `id` of the request.\n\nLet’s refactor our `getinfo` function to include some error handling, and to return the actual data we care about in the case of an error-free response:\n\n\twith url <- Application.get_env(:hello_bitcoin, :bitcoin_url),\n\t     command <- %{jsonrpc: \"1.0\", method: \"getinfo\", params: []},\n\t     {:ok, body} <- Poison.encode(command),\n\t     {:ok, response} <- HTTPoison.post(url, body),\n\t     {:ok, metadata} <- Poison.decode(response.body),\n\t     %{\"error\" => nil, \"result\" => result} <- metadata do\n\t  result\n\telse\n\t  %{\"error\" => reason} -> {:error, reason}\n\t  error -> error\n\tend\n&nbsp;\nNow our `getinfo` function will return an `{:ok, result}` tuple containing the result of our `getinfo` RPC call if everything goes well. In the case of an error we’ll receive an `{:error, reason}` tuple, explaining the failure.\n\n## Generalizing Commands\n&nbsp;\nWe could implement another Bitcoin RPC command, like `getblockhash`, in a nearly identical fashion:\n\n\tdef getblockhash(index) do\n\t  with url <- Application.get_env(:hello_bitcoin, :bitcoin_url),\n\t       command <- %{jsonrpc: \"1.0\", method: \"getblockhash\", params: [index]},\n\t       {:ok, body} <- Poison.encode(command),\n\t       {:ok, response} <- HTTPoison.post(url, body),\n\t       {:ok, metadata} <- Poison.decode(response.body),\n\t       %{\"error\" => nil, \"result\" => result} <- metadata do\n\t    {:ok, result}\n\t  else\n\t    %{\"error\" => reason} -> {:error, reason}\n\t    error -> error\n\t  end\n\tend\n&nbsp;\nCalling our new `getblockhash` with an index of `0` gives us [the hash of the Bitcoin genesis block](https://en.bitcoin.it/wiki/Genesis_block), as we would expect.\n\n\tHelloBitcoin.getblockhash(0)\n\n\t{:ok, \"000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f\"}\n&nbsp;\nWhile it’s great that this works, you’ll notice that there’s a huge amount of code duplication going on here. Our `getblockhash` function is nearly identical to our `getinfo` function.\n\nLet’s abstract out the common functionality into a new `bitcoin_rpc` helper function:\n\n\tdefp bitcoin_rpc(method, params \\\\ []) do\n\t  with url <- Application.get_env(:hello_bitcoin, :bitcoin_url),\n\t       command <- %{jsonrpc: \"1.0\", method: method, params: params},\n\t       {:ok, body} <- Poison.encode(command),\n\t       {:ok, response} <- HTTPoison.post(url, body),\n\t       {:ok, metadata} <- Poison.decode(response.body),\n\t       %{\"error\" => nil, \"result\" => result} <- metadata do\n\t    {:ok, result}\n\t  else\n\t    %{\"error\" => reason} -> {:error, reason}\n\t    error -> error\n\t  end\n\tend\n&nbsp;\nNow we can redefine our `getinfo` and `getblockhash` functions in terms of this new `bitcoin_rpc` helper function:\n\n\tdef getinfo, do: bitcoin_rpc(\"getinfo\")\n\n\tdef getblockhash(index), do: bitcoin_rpc(\"getblockhash\", [index])\n&nbsp;\nOur `bitcoin_rpc` now acts as a fully functional and complete Bitcoin RPC interface. We can easily implement any of the Bitcoin RPC commands using this helper function.\n\nIf you’re curious and want to interact with a Bitcoin node yourself, the full source for this `HelloBitcoin` project is [available on GitHub](https://github.com/pcorey/hello_bitcoin).\n\n## Wrap Up\n&nbsp;\nIn hindsight, this was a long article explaining a relatively simple idea. The Bitcoin full node software exposes a JSON-RPC interface that can easily be accessed by your favorite language or stack, such as Elixir.\n\nI’m incredibly excited about Bitcoin development, and I’m planning on spending more time diving deeper into this world in the future.\n\nIf you’re interested in the technical ideas behind Bitcoin, or are interested in Bitcoin development, I highly recommend you read [Mastering Bitcoin](https://www.amazon.com/gp/product/1449374042/ref=as_li_qf_sp_asin_il_tl?ie=UTF8&tag=east5th-20&camp=1789&creative=9325&linkCode=as2&creativeASIN=1449374042&linkId=23b0becd76b444f25a75a3143f5d253f) (affiliate link).",
      "json_metadata": "{\"tags\":[\"bitcoin\",\"programming\",\"cryptocurrency\"],\"links\":[\"https://bitcoin.org/en/\",\"https://www.amazon.com/gp/product/1449374042/ref=as_li_qf_sp_asin_il_tl?ie=UTF8&tag=east5th-20&camp=1789&creative=9325&linkCode=as2&creativeASIN=1449374042&linkId=23b0becd76b444f25a75a3143f5d253f\",\"https://elixir-lang.org/\",\"https://en.bitcoin.it/wiki/API_reference_(JSON-RPC)\",\"https://bitcoin.org/en/full-node\",\"https://bitcoin.org/en/full-node#osx-daemon\",\"https://github.com/bitcoin/bitcoin/blob/master/src/bitcoin-cli.cpp#L194-L285\",\"https://en.bitcoin.it/wiki/Original_Bitcoin_client/API_calls_list\",\"https://bitcoin.stackexchange.com/a/19839\",\"https://hex.pm/packages/poison\",\"https://hex.pm/packages/httpoison\",\"https://en.bitcoin.it/wiki/Genesis_block\",\"https://github.com/pcorey/hello_bitcoin\"],\"app\":\"steemit/0.1\",\"format\":\"markdown\"}"
    }
  ]
}

{
  "trx_id": "27255318f1c987e123c95a338c5211759f018031",
  "block": 14860773,
  "trx_in_block": 48,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-08-24T17:30:45",
  "op": [
    "claim_reward_balance",
    {
      "account": "petecorey",
      "reward_steem": "0.000 STEEM",
      "reward_sbd": "0.000 SBD",
      "reward_vests": "4.129204 VESTS"
    }
  ]
}

{
  "trx_id": "0000000000000000000000000000000000000000",
  "block": 14397404,
  "trx_in_block": 4294967295,
  "op_in_trx": 0,
  "virtual_op": 13,
  "timestamp": "2017-08-08T14:32:09",
  "op": [
    "curation_reward",
    {
      "curator": "petecorey",
      "reward": "4.129204 VESTS",
      "comment_author": "boxxa",
      "comment_permlink": "hello-steemit"
    }
  ]
}

{
  "trx_id": "fd4741214173789adf6712cac24a1765113af9f0",
  "block": 14196377,
  "trx_in_block": 8,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-08-01T14:43:03",
  "op": [
    "custom_json",
    {
      "required_auths": [],
      "required_posting_auths": [
        "petecorey"
      ],
      "id": "follow",
      "json": "[\"follow\",{\"follower\":\"petecorey\",\"following\":\"boxxa\",\"what\":[\"blog\"]}]"
    }
  ]
}

{
  "trx_id": "48c3d11db441859ea4cdb76e7c8d5f5fbf197d5c",
  "block": 14196375,
  "trx_in_block": 4,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-08-01T14:42:57",
  "op": [
    "vote",
    {
      "voter": "petecorey",
      "author": "boxxa",
      "permlink": "hello-steemit",
      "weight": 10000
    }
  ]
}

{
  "trx_id": "45663c9bd2ef9c17602cb0849c6048f9e2a7301c",
  "block": 13682123,
  "trx_in_block": 13,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-07-14T17:50:18",
  "op": [
    "vote",
    {
      "voter": "lakshya",
      "author": "petecorey",
      "permlink": "detecting-nosql-injection",
      "weight": 0
    }
  ]
}

{
  "trx_id": "0817c65cd6d63fceb0b3355315e706a77fb2aa9f",
  "block": 13682119,
  "trx_in_block": 15,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-07-14T17:50:06",
  "op": [
    "vote",
    {
      "voter": "lakshya",
      "author": "petecorey",
      "permlink": "detecting-nosql-injection",
      "weight": 10000
    }
  ]
}

{
  "trx_id": "e83f7c16cda621e756540306fc6be4c3f1e82f2e",
  "block": 13681930,
  "trx_in_block": 17,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-07-14T17:40:39",
  "op": [
    "vote",
    {
      "voter": "xochicotta",
      "author": "petecorey",
      "permlink": "detecting-nosql-injection",
      "weight": 10000
    }
  ]
}

{
  "trx_id": "a46d0686fcf533d89b3d2489b11c3c925135386e",
  "block": 13681923,
  "trx_in_block": 18,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-07-14T17:40:18",
  "op": [
    "comment",
    {
      "parent_author": "petecorey",
      "parent_permlink": "detecting-nosql-injection",
      "author": "cheetah",
      "permlink": "cheetah-re-petecoreydetecting-nosql-injection",
      "title": "",
      "body": "Hi! I am a robot. I just upvoted you! I found similar content that readers might be interested in:\nhttps://medium.com/@east5th/detecting-nosql-injection-f937e7957658",
      "json_metadata": ""
    }
  ]
}

{
  "trx_id": "4d1fcacc2125674cb6cffacf196f2edc7d681f04",
  "block": 13681922,
  "trx_in_block": 2,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-07-14T17:40:15",
  "op": [
    "vote",
    {
      "voter": "cheetah",
      "author": "petecorey",
      "permlink": "detecting-nosql-injection",
      "weight": 100
    }
  ]
}

{
  "trx_id": "ee847fdc887fb938b832fe7a552fc3ef222d4577",
  "block": 13681899,
  "trx_in_block": 14,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-07-14T17:39:06",
  "op": [
    "comment",
    {
      "parent_author": "",
      "parent_permlink": "programming",
      "author": "petecorey",
      "permlink": "detecting-nosql-injection",
      "title": "Detecting NoSQL Injection",
      "body": "# Detecting NoSQL Injection\n\n \n\nThe entire premise behind my latest project, [Inject Detect](http://www.injectdetect.com/), is that [NoSQL Injection attacks](http://www.east5th.co/blog/2017/07/03/what-is-nosql-injection/) can be detected in real-time as they’re being carried out against your application.\n\nBut how?\n\nIn this article, I’ll break down the strategy I’m using for detecting NoSQL Injection in MongoDB-based web applications.\n\nAt a high level, the idea is to build up a set of expected queries an application is known to make, and to use that set to detect unexpected queries that might be to result of a NoSQL Injection attack.\n\nLet’s dig into the details.\n\n \n\n## Expected Queries\n\n \n\nEvery web application has a finite number of queries it can be expected to make throughout the course of its life.\n\nFor example, a shopping cart application might query for single orders by `_id`:\n\n\tOrders.findOne(orderId);\n\n \n\nSimilarly, it might query for a number of orders created in the past three days:\n\n\tOrders.find({createdAt: {$gte: moment().subtract(3, \"days\").toDate()}});\n\n \n\nThese queries aren’t limited to fetching data. When a user “deletes” an order, the application may want to set a flag on the order in question:\n\n\tOrders.update({userId: this.userId}, {$set: {deleted: true}});\n\n \n\nEach of these individual queries can be generalized based on the shape of the query and the type of data passed into it.\n\nFor example, we expect the `Orders.findOne` query to always be called with a String. Similarly, we expect the `Orders.find` query to be passed a Date for the `createdAt` comparison. Lastly, the `Orders.update` query should always be passed a String as the `userId`.\n\n\tOrders.findOne({_id: String});\n\t\n\tOrders.find({createdAt: {$gte: Date}});\n\t\n\tOrders.update({userId: String}, ...);\n\n \n\nAn application might make thousands of queries per day, but each query will match one of these three generalized query patterns.\n\n \n\n## Unexpected Queries\n\n \n\nIf our application makes a query that does not fall into this set of expected queries, we’re faced with one of two possibilities:\n\n1. We left a query out of our set of expected queries.\n2. Our application is vulnerable to a NoSQL Injection vulnerability.\n\nImagine our application makes the following query:\n\n\tOrders.findOne({_id: { $gte: \"\" }});\n\n \n\nA query of this pattern (`Orders.findOne({_id: {$gte: String}})`) doesn’t exist in our set of expected queries. This means that this is either an expected query that we missed, or our application is being exploited.\n\nIt’s unlikely that our application is trying to find a single `Order` who’s `_id` is greater than or equal to an empty string.\n\nIn this case, it’s far more likely that someone is exploiting our `Orders.findOne({_id: String})` query and passing in an `orderId` containing a MongoDB query operator (`{$gte: \"\"}`) rather than an expected String.\n\nWe’ve detected NoSL Injection!\n\nBy watching for queries that fall outside our set of expected queries, we can detect NoSQL Injection as it happens.\n\n \n\n## Similar Expected Queries\n\n \n\nBasing our NoSQL Injection detection strategy around expected and unexpected queries has an added bonus.\n\nBecause we have a set of all expected queries for a given application, unexpected queries that are the result of NoSQL Injection attacks can often be linked back to the query being exploited.\n\nTo illustrate, in our previous example we detected an unexpected query against our application:\n\n\tOrders.findOne({_id: {$gte: \"\"}});\n\n \n\nInspecting our set of expected queries, it’s obvious that the most similar expected query is the `Orders.findOne` query:\n\n\tOrders.findOne({_id: String});\n\n \n\nAs the application owner, we know that we need to enforce more stringent checks on the type of the provided `orderId`.\n\nBased on this information, an application owner or developer can deduce which specific query is being exploited within their application and [respond quickly with a fix](http://www.east5th.co/blog/2017/07/03/what-is-nosql-injection/#how-do-you-prevent-nosql-injection).\n\n \n\n## Final Thoughts\n\n \n\nProgress on [Inject Detect](http://www.injectdetect.com/) continues to move along at a steady pace.\n\nOnce finished, Inject Detect will automatically apply this type of real-time NoSQL Injection detection to every query made by your Meteor application.\n\nIf you’re interested in learning more, be sure to sign up for the [Inject Detect newsletter](http://www.injectdetect.com/#sign-up).",
      "json_metadata": "{\"tags\":[\"programming\",\"web\",\"security\",\"nosql\",\"injection\"],\"links\":[\"http://www.injectdetect.com/\",\"http://www.east5th.co/blog/2017/07/03/what-is-nosql-injection/\",\"http://www.east5th.co/blog/2017/07/03/what-is-nosql-injection/#how-do-you-prevent-nosql-injection\",\"http://www.injectdetect.com/#sign-up\"],\"app\":\"steemit/0.1\",\"format\":\"markdown\"}"
    }
  ]
}

{
  "trx_id": "f57597e55e95a2d83e8a91da2dd28857a285c2bc",
  "block": 13476561,
  "trx_in_block": 0,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-07-07T14:19:57",
  "op": [
    "claim_reward_balance",
    {
      "account": "petecorey",
      "reward_steem": "0.000 STEEM",
      "reward_sbd": "0.012 SBD",
      "reward_vests": "14.477589 VESTS"
    }
  ]
}

{
  "trx_id": "0000000000000000000000000000000000000000",
  "block": 13447652,
  "trx_in_block": 4294967295,
  "op_in_trx": 0,
  "virtual_op": 13,
  "timestamp": "2017-07-06T14:10:18",
  "op": [
    "author_reward",
    {
      "author": "petecorey",
      "permlink": "distributed-systems-are-hard",
      "sbd_payout": "0.012 SBD",
      "steem_payout": "0.000 STEEM",
      "vesting_payout": "14.477589 VESTS"
    }
  ]
}

{
  "trx_id": "c5e8bf32fb1680752c67b805a303ef0c0a8412ef",
  "block": 13287913,
  "trx_in_block": 7,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-07-01T00:39:48",
  "op": [
    "claim_reward_balance",
    {
      "account": "petecorey",
      "reward_steem": "0.000 STEEM",
      "reward_sbd": "0.009 SBD",
      "reward_vests": "12.415216 VESTS"
    }
  ]
}

{
  "trx_id": "61db045a840a1c797a59144e1c20832737db8224",
  "block": 13287901,
  "trx_in_block": 27,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-07-01T00:39:12",
  "op": [
    "vote",
    {
      "voter": "petecorey",
      "author": "steemitboard",
      "permlink": "steemitboard-notify-petecorey-20170630t161914000z",
      "weight": 10000
    }
  ]
}

{
  "trx_id": "78e20d0c2e152993cbb2907b34c3799d9508f4d1",
  "block": 13277906,
  "trx_in_block": 1,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-06-30T16:19:15",
  "op": [
    "comment",
    {
      "parent_author": "petecorey",
      "parent_permlink": "the-ecstasy-of-testing",
      "author": "steemitboard",
      "permlink": "steemitboard-notify-petecorey-20170630t161914000z",
      "title": "",
      "body": "Congratulations @petecorey! You have completed some achievement on Steemit and have been rewarded with new badge(s) :\n\n[![](https://steemitimages.com/70x80/http://steemitboard.com/notifications/voted.png)](http://steemitboard.com/@petecorey) Award for the number of upvotes received\n[![](https://steemitimages.com/70x80/http://steemitboard.com/notifications/votes.png)](http://steemitboard.com/@petecorey) Award for the number of upvotes\n\nClick on any badge to view your own Board of Honnor on SteemitBoard.\nFor more information about SteemitBoard, click [here](https://steemit.com/@steemitboard)\n\nIf you no longer want to receive notifications, reply to this comment with the word `STOP`\n\nBy upvoting this notification, you can help all Steemit users. Learn how [here](https://steemit.com/steemitboard/@steemitboard/http-i-cubeupload-com-7ciqeo-png)!",
      "json_metadata": "{\"image\":[\"https://steemitboard.com/img/notifications.png\"]}"
    }
  ]
}

{
  "trx_id": "5bd23033c03d303207ed183e3cd3e1d00c8036a3",
  "block": 13252995,
  "trx_in_block": 17,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-06-29T19:33:15",
  "op": [
    "vote",
    {
      "voter": "petecorey",
      "author": "petecorey",
      "permlink": "the-ecstasy-of-testing",
      "weight": 10000
    }
  ]
}

{
  "trx_id": "5bd23033c03d303207ed183e3cd3e1d00c8036a3",
  "block": 13252995,
  "trx_in_block": 17,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-06-29T19:33:15",
  "op": [
    "comment",
    {
      "parent_author": "",
      "parent_permlink": "programming",
      "author": "petecorey",
      "permlink": "the-ecstasy-of-testing",
      "title": "The Ecstasy of Testing",
      "body": "# The Ecstasy of Testing\n\n  \n\n`describe(\"The ecstasy of testing\", function() {`\n\nYou dive in, equipped with nothing more than a creeping dissatisfaction and a passing test suite.\n\nAs the darkness of uncertainty begins to envelop you, the shining green light of your passing suite fills you with unbounded confidence. Fearlessly, you make your first refactor. Vast swaths of code fall away before you, subjected to the annals of git history. Intricate towers of loosely coupled components assemble themselves upon the foundations of your assumptions and beliefs.\n\nIn your gut, you feel that it’s ready. You hit save. Your waiting tests eagerly devour your creation, but something is wrong. The suite is failing. Exceptions smoulder red with roars of unmet assertions. You realize that you've missed an entire set of edge cases. Your assumptions were unfounded, and were it not for the meticulous diligence of your test suite, devastating. Cries of what could have been are drowned out by a torrent of mechanical clicks and clacks as your fingers deftly recover from your oversight.\n\nOnce again you set the suite into motion. Between the hushed sounds of your bated breath you can nearly hear the whirring cogs of your test harness as it churns through vast permutations of possibilities and potentialities. Finally, the gears come to rest.\n\nThe screen glows green in front of you.\n\n_Your tests are passing._\n\n___It's finished.___\n\n`});`",
      "json_metadata": "{\"tags\":[\"programming\",\"testing\",\"software\",\"quality\"],\"app\":\"steemit/0.1\",\"format\":\"markdown\"}"
    }
  ]
}

{
  "trx_id": "d5ca18475db46f1d40960555e81ab327397a9d4a",
  "block": 13247920,
  "trx_in_block": 0,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-06-29T15:19:18",
  "op": [
    "vote",
    {
      "voter": "petecorey",
      "author": "cheetah",
      "permlink": "cheetah-re-petecoreydistributed-systems-are-hard",
      "weight": 10000
    }
  ]
}

{
  "trx_id": "9fbd96b2baefcdb749e196f3206e0719a663c82e",
  "block": 13246573,
  "trx_in_block": 23,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-06-29T14:11:45",
  "op": [
    "vote",
    {
      "voter": "speakerhornet",
      "author": "petecorey",
      "permlink": "distributed-systems-are-hard",
      "weight": 10000
    }
  ]
}

{
  "trx_id": "d07cd762b0eab099cda46df93b62fba381f48349",
  "block": 13246572,
  "trx_in_block": 8,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-06-29T14:11:42",
  "op": [
    "comment",
    {
      "parent_author": "petecorey",
      "parent_permlink": "distributed-systems-are-hard",
      "author": "cheetah",
      "permlink": "cheetah-re-petecoreydistributed-systems-are-hard",
      "title": "",
      "body": "Hi! I am a robot. I just upvoted you! I found similar content that readers might be interested in:\nhttp://www.east5th.co/blog/2017/06/26/distributed-systems-are-hard/",
      "json_metadata": ""
    }
  ]
}

{
  "trx_id": "c3f9c37737e95c3df9d8eec13f8a76034bcdf8ab",
  "block": 13246570,
  "trx_in_block": 19,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-06-29T14:11:36",
  "op": [
    "vote",
    {
      "voter": "cheetah",
      "author": "petecorey",
      "permlink": "distributed-systems-are-hard",
      "weight": 100
    }
  ]
}

{
  "trx_id": "0515be12dadb7373cff3247ec5bdc88b1357359b",
  "block": 13246544,
  "trx_in_block": 15,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-06-29T14:10:18",
  "op": [
    "vote",
    {
      "voter": "petecorey",
      "author": "petecorey",
      "permlink": "distributed-systems-are-hard",
      "weight": 10000
    }
  ]
}

{
  "trx_id": "0515be12dadb7373cff3247ec5bdc88b1357359b",
  "block": 13246544,
  "trx_in_block": 15,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-06-29T14:10:18",
  "op": [
    "comment",
    {
      "parent_author": "",
      "parent_permlink": "programming",
      "author": "petecorey",
      "permlink": "distributed-systems-are-hard",
      "title": "Distributed Systems Are Hard",
      "body": "# Distributed Systems Are Hard\n\nAs I dive deeper and deeper into the world of [Elixir](https://elixir-lang.org/) and distributed systems in general, I’ve been falling deeper and deeper into a personal crisis.\n\nI’ve been slowly coming to the realization that just about every production system I’ve worked on or built throughout my career is broken in one way or another.\n\nDistributed systems are hard.\n\n## Horizontal Scaling is Easy, Right?\n\nIn the past, my solution to the problem of scale has always been to scale horizontally. By “scale horizontally”, I mean spinning up multiple instances of your server processes, either across multiple CPUs, or multiple machines, and distributing traffic between them.\n\nAs long as my server application doesn’t persist in-memory state across sessions, or persist anything to disk, it’s fair game for horizontal scaling. For the most part, this kind of shoot-from-the-hip horizontal scaling works fairly well…\n\nUntil it doesn’t.\n\nWithout careful consideration and deliberate design, “split it and forget it” scaling will eventually fail. It may not fail catastrophically - in fact, it will most likely fail in subtle, nuanced ways. But it will always fail.\n\n> This is the way the world ends\n> Not with a bang but a whimper.\n\nLet’s take a look at how this type of scaling can break down and introduce [heisenbugs](https://en.wikipedia.org/wiki/Heisenbug) into your system.\n\n## Scaling in Action\n\nFor the sake of discussion, imagine that we’re building a web application that groups users into teams. A rule, or invariant, of our system is that a user can only be assigned to a single team at a time.\n\nOur system enforces this rule by checking if a user already belongs to a team before adding them to another:\n\n\tfunction addUserToTeam(userId, teamId) {\n\t    if (Teams.findOne({ userIds: userId })) {\n\t        throw new Error(\"Already on a team!\");\n\t    }\n\t    Teams.update({ _id: teamId }, { $push: { userIds: userId } });\n\t}\n\nThis seems relatively straight-forward, and has worked beautifully in our small closed-beta trials.\n\nGreat! Over time, our Team Joiner™ application becomes very popular.\n\n![Team Joiner on Product Hunt](https://s3-us-west-1.amazonaws.com/www.east5th.co/img/team-joiner.png)\n\nTo meet the ever growing demand of new users wanting to join teams, we begin horizontally scaling our application by spinning up more instances of our server. However, as we add more servers, mysterious bugs begin to crop up…\n\nUsers are somehow, under unknown circumstances, joining multiple teams. That was supposed to be a premium feature!\n\n## With Our Powers Combined\n\nThe root of the problem stems from the fact that we have two (or more) instances of our server process running in parallel, without accounting for the existence of the other processes.\n\nImagine a scenario where a user, Sue, attempts to join Team A. Simultaneously, an admin user, John, notices that Sue isn’t on a team and decides to help by assigning her to Team B.\n\nSue’s request is handled entirely by Server A, and John’s request is handled entirely by Server B.\n\nServer A begins by checking if Sue is on a team. She is not. Just after that, Server B also checks if Sue is on a team. She is not. At this point, both servers think they’re in the clear to add Sue to their respective team. Server A assigns Sue to Team A, fulfilling her request. Meanwhile, Server B assigns Sue to Team B, fulfilling John’s request.\n\n![Diagram of conflict between Server A and Server B.](https://s3-us-west-1.amazonaws.com/www.east5th.co/img/team-conflict.png)\n\nInterestingly, both servers do their jobs flawlessly individually, while their powers combined put the system in an invalid, unpredictable, and potentially unrecoverable state.\n\n---- \n\nThe issue here is that between the point in time when Server B verifies that Sue is not on a team and the point when it assigns her to Team B, the state of the system changes.\n\nServer B carries out its database update operating under the assumptions of old,  stale data. The server process isn’t properly designed to handle, or even recognize these types of conflicting updates.\n\nInterestingly (and horrifyingly), this isn’t the only type of bug that can result from this type of haphazard scaling.\n\nCheck out the beginning of [Nathan Herald’s talk from this year’s ElixirConf EU](https://www.youtube.com/watch?v=SDKiLO2XwIs) to hear about all of the fantastic ways that distributed systems can fail.\n\n## Handling Conflicts\n\nWhile this specific problem is somewhat contrived and could be easily fixed by a database schema that more accurately reflects the problem we’re trying to solve (by keeping `teamId` on the user document), it serves as a good platform to discuss the larger issue.\n\nDistributed systems are hard.\n\nWhen building distributed systems, you need to be prepared to be working with data that may be inconsistent or outdated. Conflicts should be an expected outcome that are designed into the system and strategically planned for.\n\nThis is part of the reason I’ve [gravitated towards an Event Sourcing approach](http://www.east5th.co/blog/2017/06/19/genservers-and-memory-images-a-match-made-in-heaven/#backed-by-an-event-log) for my latest project, [Inject Detect](http://www.injectdetect.com/).\n\nEvents can be ordered sequentially in your database, and you can make assertions (with the help of database indexing) that the event you’re inserting immediately follows the last event you’ve seen.\n\nWe’ll dive into the details surrounding this type of solution in future posts.\n\n## Final Thoughts\n\nWrapping up, I feel like this article ranks high in fear-mongering and low in actionable value. That definitely isn’t my intention.\n\nMy goal is to show that working with distributed systems is unexpectedly hard. The moment you add a second CPU or spin up a new server instance, you’re entering a brave new (but really, not so new) world of computing that requires you to more deeply consider every line of code you write.\n\nI encourage you to re-examine projects and code you’ve written that exist in a distributed environment. Have you ever experienced strange bugs that you can’t explain? Are there any race conditions lurking there that you’ve never considered?\n\nIs your current application ready to be scaled horizontally? Are you sure?\n\nIn the future, I hope to write more actionable articles about solving these kinds of problems. Stay tuned for future posts on how Event Sourcing can be used to write robust, conflict-free distributed systems!",
      "json_metadata": "{\"tags\":[\"programming\",\"software\",\"server\",\"scaling\",\"javascript\"],\"image\":[\"https://s3-us-west-1.amazonaws.com/www.east5th.co/img/team-joiner.png\",\"https://s3-us-west-1.amazonaws.com/www.east5th.co/img/team-conflict.png\"],\"links\":[\"https://elixir-lang.org/\",\"https://en.wikipedia.org/wiki/Heisenbug\",\"https://www.youtube.com/watch?v=SDKiLO2XwIs\",\"http://www.east5th.co/blog/2017/06/19/genservers-and-memory-images-a-match-made-in-heaven/#backed-by-an-event-log\",\"http://www.injectdetect.com/\"],\"app\":\"steemit/0.1\",\"format\":\"markdown\"}"
    }
  ]
}

{
  "trx_id": "0000000000000000000000000000000000000000",
  "block": 13194215,
  "trx_in_block": 4294967295,
  "op_in_trx": 0,
  "virtual_op": 12,
  "timestamp": "2017-06-27T18:19:51",
  "op": [
    "author_reward",
    {
      "author": "petecorey",
      "permlink": "genservers-and-memory-images-a-match-made-in-heaven",
      "sbd_payout": "0.009 SBD",
      "steem_payout": "0.000 STEEM",
      "vesting_payout": "12.415216 VESTS"
    }
  ]
}

{
  "trx_id": "2b8c17ddabd8874bfd5c3477d27376e79e51b58f",
  "block": 13078198,
  "trx_in_block": 1,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-06-23T17:34:45",
  "op": [
    "vote",
    {
      "voter": "petecorey",
      "author": "twitterbot",
      "permlink": "re-have-you-tried-just-using-a-function-20170623t171712",
      "weight": 10000
    }
  ]
}

{
  "trx_id": "d3f017db3b1bc8183d9ecf6865f42bbc87265027",
  "block": 13077847,
  "trx_in_block": 15,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-06-23T17:17:12",
  "op": [
    "comment",
    {
      "parent_author": "petecorey",
      "parent_permlink": "have-you-tried-just-using-a-function",
      "author": "twitterbot",
      "permlink": "re-have-you-tried-just-using-a-function-20170623t171712",
      "title": "",
      "body": "### ![ZachAMcCoy](https://pbs.twimg.com/profile_images/808732663204806656/-MF2TOqw_normal.jpg) **[Zach McCoy](https://twitter.com/@ZachAMcCoy/status/839980751529644033)** tweeted @ 09 Mar 2017 - 23:26 UTC\n\n> Have you tried just using a function?\n\n\n###### *Disclaimer: I am just a bot trying to be helpful.*",
      "json_metadata": ""
    }
  ]
}

{
  "trx_id": "1a1fe40b448e70ae4afde5191fc903637d55b961",
  "block": 13077824,
  "trx_in_block": 15,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-06-23T17:16:03",
  "op": [
    "vote",
    {
      "voter": "petecorey",
      "author": "petecorey",
      "permlink": "have-you-tried-just-using-a-function",
      "weight": 10000
    }
  ]
}

{
  "trx_id": "1a1fe40b448e70ae4afde5191fc903637d55b961",
  "block": 13077824,
  "trx_in_block": 15,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-06-23T17:16:03",
  "op": [
    "comment",
    {
      "parent_author": "",
      "parent_permlink": "programming",
      "author": "petecorey",
      "permlink": "have-you-tried-just-using-a-function",
      "title": "Have You Tried Just Using a Function?",
      "body": "# Have You Tried Just Using a Function?\n\nLast month I read Saša Jurić’s [To Spawn, or Not to Spawn](http://theerlangelist.com/article/spawn_or_not) article and its been lurking in my subconscious ever since.\n\nI was recently working on [the command handler and event sourcing system](http://www.east5th.co/blog/2017/05/01/inject-detect-progress-report/#back-end-progress) that drives my new project, [Inject Detect](http://www.injectdetect.com/), and this exact topic reared its head. I realized that I had been overcomplicating the project with exessive usage of Elixir processes.\n\nRefactoring my command handler from a process into simple functions hugely simplified the application, and opened the doors for a new set of functionality I wanted to implement.\n\n## The Command Handler Process\n\nFor a high level overview, a “command” in Inject Detect represents something you want to do in the system, like requesting a new sign-in token for a user (`RequestSignInToken`), or ingesting a batch of queries from a user’s application (`IngestQueries`).\n\nCommands are “handled” by passing them to the command handler:\n\n\t%IngestQueries{application_id: application.id, queries: queries}\n\t|> CommandHandler.handle(command)\n\nThe job of the command handler is to determine if the command is allowable based on the state of the system and the current user’s authorizations. If valid, the command being handled (`IngestQueries` in this case) will produce a list of events (such as `IngestedQuery` and `IngestedUnexpectedQuery`). These events are saved to the database, and a handful of “event listeners” are notified.\n\n## Command Handler as a Process\n\nOriginally, the `CommandHandler` was implemented as a [GenServer](https://hexdocs.pm/elixir/GenServer.html)-based [Elixir](https://elixir-lang.org/) process. The call to `CommandHandler.handle` triggered a `GenServer.call` to the `CommandHandler` process:\n\n\tdef handle(command, context \\\\ %{}) do\n\t  GenServer.call(__MODULE__, {:handle, command, context})\n\tend\n\nThe corresponding `handle_call` callback would handle the command, store the resulting events, and synchronously notify any interested listeners:\n\n\tdef handle_call({:handle, command, context}, _, []) do\n\t  with {:ok, events, context} <- handle_command(command, context),\n\t       {:ok, _}               <- store_events(events),\n\t  do\n\t    notify_listeners(events, context)\n\t    {:reply, {:ok, context}, []}\n\t  else\n\t    error -> {:reply, error, []}\n\t  end\n\tend\n\n## Triggering Commands from Listeners\n\nFor several weeks, this solution worked just fine. It wasn’t until I started adding more complex event listeners that I ran into real issues.\n\nI mentioned earlier that event listeners are notified whenever an event is produced by a command. In some cases, these listeners may want to fire off a new command. For instance, when an `IngestedUnexpectedQuery` event is fired, a listener may want to execute a `SendUnexpectedEmail` command.\n\nImplementing this feature blew up in my face.\n\nBecause listeners are called synchronously from my `CommandHandler.handle` function, another call to `CommandHandler.handle` from within a listener would result in a GenServer timeout.\n\nThe first call to `CommandHandler.handle ` won’t reply until the second `CommandHandler.handle ` call is finished, but the second `CommandHandler.handle` call won’t be processed until the first call finishes. The second call will wait until it hits its timeout threshold and eventually fail.\n\nWe’ve hit a deadlock.\n\nThe only way to handle this situation would be to execute either the second call to `CommandHandler.handle`, or the entire listener function within an unsupervised, asynchronous process:\n\n\tTask.start(fn -> \n\t  %SendUnexpectedEmail{...}\n\t  |> CommandHandler.handle\n\tend)\n\nI wasn’t willing to go down this road due to testing difficulties and a general distrust of unsupervised children.\n\n## Command Handler as a Function\n\nAfter mulling over my deadlock problem, the solution slapped me in the face.\n\nThe functionality of the command handler could be entirely implemented as a module of simple functions. No process or GenServer required.\n\nA quick refactor led me to this solution:\n\n\tdef handle(command, context, listeners) do\n\t  with {:ok, events, context} <- handle_command(command, context),\n\t       {:ok, _}               <- store_events(events)\n\t  do\n\t    notify_listeners(events, context, listeners)\n\t    {:ok, context}\n\t  end\n\tend\n\nAfter the refactor, a synchronously called event listener can recursively call `CommandHandler.handle` to handle any follow-up commands it wants to execute.\n\nPerfect.\n\n## Have You Tried Just Using a Function?\n\nIn hindsight, I had no particular reason for implementing the `CommandHandler` module as a GenServer. It managed no state and had no specific concurrency concerns that demanded the use of a process.\n\nWhen given a hammer, everything starts to look like a nail.\n\nRemember to use the right tool for the job. In many cases, the right tool is the simplest tool. Often, the simplest tool for the job is to [just use a function](https://twitter.com/ZachAMcCoy/status/839980751529644033).",
      "json_metadata": "{\"tags\":[\"programming\",\"elixir\",\"functional\",\"otp\",\"architecture\"],\"links\":[\"http://theerlangelist.com/article/spawn_or_not\",\"http://www.east5th.co/blog/2017/05/01/inject-detect-progress-report/#back-end-progress\",\"http://www.injectdetect.com/\",\"https://hexdocs.pm/elixir/GenServer.html\",\"https://elixir-lang.org/\",\"https://twitter.com/ZachAMcCoy/status/839980751529644033\"],\"app\":\"steemit/0.1\",\"format\":\"markdown\"}"
    }
  ]
}

{
  "trx_id": "f801727bb8f29fa0aebb679bb5d94fd12c7c0239",
  "block": 13049145,
  "trx_in_block": 29,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-06-22T17:21:30",
  "op": [
    "claim_reward_balance",
    {
      "account": "petecorey",
      "reward_steem": "0.000 STEEM",
      "reward_sbd": "0.030 SBD",
      "reward_vests": "28.978114 VESTS"
    }
  ]
}

{
  "trx_id": "0000000000000000000000000000000000000000",
  "block": 13022226,
  "trx_in_block": 4294967295,
  "op_in_trx": 0,
  "virtual_op": 14,
  "timestamp": "2017-06-21T18:55:12",
  "op": [
    "author_reward",
    {
      "author": "petecorey",
      "permlink": "graphql-nosql-injection-through-json-types",
      "sbd_payout": "0.030 SBD",
      "steem_payout": "0.000 STEEM",
      "vesting_payout": "28.978114 VESTS"
    }
  ]
}

{
  "trx_id": "b9753a0f47a88801723add1637da2b1521538128",
  "block": 12992859,
  "trx_in_block": 9,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-06-20T18:19:51",
  "op": [
    "vote",
    {
      "voter": "petecorey",
      "author": "petecorey",
      "permlink": "genservers-and-memory-images-a-match-made-in-heaven",
      "weight": 10000
    }
  ]
}

{
  "trx_id": "b9753a0f47a88801723add1637da2b1521538128",
  "block": 12992859,
  "trx_in_block": 9,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-06-20T18:19:51",
  "op": [
    "comment",
    {
      "parent_author": "",
      "parent_permlink": "programming",
      "author": "petecorey",
      "permlink": "genservers-and-memory-images-a-match-made-in-heaven",
      "title": "GenServers and Memory Images : A Match Made in Heaven",
      "body": "# GenServers and Memory Images - A Match Made in Heaven\n\nMy current project, [Inject Detect](http://www.injectdetect.com/), is being built with [Elixir](https://elixir-lang.org/) and makes heavy use of Martin Fowler-style [Memory Images](https://martinfowler.com/bliki/MemoryImage.html). After working with this setup for several months, I’ve come to realize that Elixir [GenServers](https://elixir-lang.org/getting-started/mix-otp/genserver.html) and a Memory Image architecture are a match made in heaven.\n\nLet’s dive into what Memory Images are, and why GenServers are the perfect tool for building out a Memory Image in your application.\n\n## What is a Memory Image?\n\nIn my opinion, the best introduction to the Memory Image concept is [Martin Fowler’s article on the subject](https://martinfowler.com/bliki/MemoryImage.html). If you haven’t, be sure to read through the article.\n\nFor brevity, I’ll try to summarize as quickly as possible. Martin comments that most developers’ first question when starting a new project is, “what database will I use?” Unfortunately, answering this question requires many upfront decisions about things like data shape and even usage patterns that are often unknowable upfront.\n\nMartin flips the question on its head. Instead of asking which database you should use, he suggests you ask yourself, “do I need a database at all?”\n\nMind blown.\n\nThe idea of a Memory Image is to keep the entire state of your application entirely within your server’s memory, rather than keeping it in a database. At first, this seems absurd. In reality, it actually works very well for many projects.\n\nI’ll defer an explanation of the pros, cons, and my experiences with Memory Images to a later post. Instead of going down that rabbit hole, let’s take a look at how we can efficiently implement a Memory Image in Elixir!\n\n## Backed By an Event Log\n\nThe notion that a Memory Image architecture don’t use a database at all isn’t entirely true. In Inject Detect, I use a database to persist a log of events that describe all changes that have happened to the system since the beginning of time.\n\nThis event log isn’t particularly useful in its raw format. It can’t be queried in any meaningful way, and it can’t be used to make decisions about the current state of the system.\n\nTo get something more useful out of the system, the event log needs to be replayed. Each event effects the system’s state in some known way. By replaying these events and their corresponding effects in order, we can rebuild the current state of the system. We effectively reduce down all of the events in our event log into the current state of our system.\n\nThis is [Event Sourcing](https://martinfowler.com/eaaDev/EventSourcing.html).\n\n---- \n\nWe can implement this kind of simplified Event Sourced system fairly easily:\n\n\tdefmodule State do\n\t\n\t  def get do\n\t    state = InjectDetect.Model.Event\n\t    |> order_by([event], event.id)\n\t    |> InjectDetect.Repo.all\n\t    |> Enum.to_list\n\t    |> Enum.map(&(struct(String.to_atom(&1.type), &1.data))\n\t    |> Enum.reduce(%{}, &State.Reducer.apply/2)\n\t  end\n\t\n\tend\n\nEach event in our event log has a `type` field that points to a specific event struct in our application (like `SignedUp`), and a `data` field that holds a map of all the information required to replay the effects of that event on the system.\n\nFor example, a `SignedUp` event might look like this when saved to the database:\n\n\t%{id: 123, type: \"SignedUp\", data: %{\"email\" => \"[email protected]\"}}\n\nTo get the current state of the system, we grab all events in our event log, convert them into structs, and then reduce them down into a single state object by applying their changes, one after the other, using our `State.Reducer.apply`  [Elixir protocol](https://elixir-lang.org/getting-started/protocols.html) that all event structs are required to implement.\n\nWhile this is a fairly simple concept, it’s obviously inefficient. Imagine having to process your entire event log every time you want to inspect the state of your system!\n\nThere has to be a better way.\n\n## GenServer, Meet Memory Image\n\nMemory Image, meet [GenServer](https://elixir-lang.org/getting-started/mix-otp/genserver.html).\n\nRather than reprocessing our entire event log every time we want to inspect our application’s state, what if we could just keep the application state in memory?\n\nGenServers (and [Elixir processes](https://elixir-lang.org/getting-started/processes.html) in general) are excellent tools for persisting state in memory. Let’s refactor our previous solution to calculate our application’s state and then store it in memory for future use.\n\nTo manage this, our GenServer will need to store two pieces of information. It will need to store the current state of the system, and the `id` of the last event that was processed. Initially, our current application state will be an empty map, and the last `id` we’ve seen will be `0`:\n\n\t  def start_link, do:\n\t    GenServer.start_link(__MODULE__, {%{}, 0}, name: __MODULE__)\n\nNext, rather than fetching all events from our event log, we want to fetch only the events that have happened after the last event `id` that we’ve processed:\n\n\t  defp get_events_since(id) do\n\t    events = InjectDetect.Model.Event\n\t    |> where([event], event.id > ^id)\n\t    |> order_by([event], event.id)\n\t    |> InjectDetect.Repo.all\n\t    |> Enum.to_list\n\t    {convert_to_structs(events), get_last_event_id(events)}\n\t  end\n\nThis function returns a tuple of the fetched events, along with the `id` of the last event in that list.\n\nWhen `get_events_since` is first called, it will return all events currently in the event log. Any subsequent calls will only return the events that have happened after the last event we’ve processed. Because we’re storing the system’s state in our GenServer, we can apply these new events to the old state to get the new current state of the system.\n\nTying these pieces together, we get something like this:\n\n\tdefmodule State do\n\t  use GenServer\n\t\n\t  import Ecto.Query\n\t\n\t  def start_link, do: GenServer.start_link(__MODULE__, {%{}, 0}, name: __MODULE__)\n\t \n\t  def get, do: GenServer.call(__MODULE__, :get)\n\t\n\t  def convert_to_structs(events), do: Enum.map(events, &(struct(String.to_atom(&1.type), &1.data))\n\t\n\t  def get_last_event_id(id, events) do\n\t    case List.last(events) do\n\t      nil   -> id\n\t      event -> event.id\n\t    end\n\t  end\n\t\n\t  defp get_events_since(id) do\n\t    events = InjectDetect.Model.Event\n\t    |> where([event], event.id > ^id)\n\t    |> order_by([event], event.id)\n\t    |> InjectDetect.Repo.all\n\t    |> Enum.to_list\n\t    {convert_to_structs(events), get_last_event_id(id, events)}\n\t  end\n\t\n\t  def handle_call(:get, _, {state, last_id}) do\n\t    {events, last_id} = get_events_since(last_id)\n\t    state = Enum.reduce(events, state, &State.Reducer.apply/2)\n\t    {:reply, {:ok, state}, {state, last_id}}\n\t  end\n\t\n\tend\n\nAt first this solution may seem complicated, but when we break it down, there’s not a whole lot going on.\n\nOur `State` GenServer stores:\n1. The current `state` of the system.\n2. The `id` of the last event it has processed.\n\nWhenever we call `State.get`, it checks for new events in the event log and applies them, in order, to the current state. The GenServer saves this state and the `id` of the last new event and then replies with the new state.\n\nThat’s it!\n\n## Final Thoughts\n\nBuilding a Memory Image in Elixir using GenServers is a match made in heaven. When working with these tools and techniques, it honestly feels like solutions effortlessly fall into place.\n\nThe Memory Image architecture, especially when combined with Event Sourcing, perfectly lends itself to a functional approach. Additionally, using GenServers to implement these ideas opens the doors to building fast, efficient, fault-tolerant, and consistent distributed systems with ease.\n\nWhile Memory Images are an often overlooked solution to the problem of maintaining state, the flexibility and speed they bring to the table should make them serious contenders in your next project.",
      "json_metadata": "{\"tags\":[\"programming\",\"elixir\",\"otp\",\"architecture\",\"database\"],\"links\":[\"http://www.injectdetect.com/\",\"https://elixir-lang.org/\",\"https://martinfowler.com/bliki/MemoryImage.html\",\"https://elixir-lang.org/getting-started/mix-otp/genserver.html\",\"https://martinfowler.com/eaaDev/EventSourcing.html\",\"https://elixir-lang.org/getting-started/protocols.html\",\"https://elixir-lang.org/getting-started/processes.html\"],\"app\":\"steemit/0.1\",\"format\":\"markdown\"}"
    }
  ]
}

{
  "trx_id": "cf5edac229be53253806e7344c8bd7292b0b4b08",
  "block": 12841322,
  "trx_in_block": 9,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-06-15T11:50:51",
  "op": [
    "account_update",
    {
      "account": "petecorey",
      "memo_key": "STM72cUgzYnCWx1MHgTdiv12U2bZeyJCCFMY67Rpd2sK72qhJdxJ2",
      "json_metadata": "{\"profile\":{\"profile_image\":\"http://www.east5th.co/img/portrait-150x150.png\",\"name\":\"Pete Corey\",\"website\":\"http://www.east5th.co/blog/\"}}"
    }
  ]
}

{
  "trx_id": "60999f383f67b1dc74d0bd8e8042f9a9bcb22dcc",
  "block": 12826984,
  "trx_in_block": 10,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-06-14T23:52:21",
  "op": [
    "vote",
    {
      "voter": "adnanrahic",
      "author": "petecorey",
      "permlink": "graphql-nosql-injection-through-json-types",
      "weight": 10000
    }
  ]
}

{
  "trx_id": "c0034fb1a59be1fa09435b973654ccaba43dbbf8",
  "block": 12824835,
  "trx_in_block": 2,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-06-14T22:04:45",
  "op": [
    "vote",
    {
      "voter": "petecorey",
      "author": "mrdlucksfitness",
      "permlink": "why-a-meat-eating-bodybuilder-went-vegan",
      "weight": 10000
    }
  ]
}

{
  "trx_id": "d95098f0a90e2f4c1198896044093766098bc274",
  "block": 12821229,
  "trx_in_block": 0,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-06-14T19:04:18",
  "op": [
    "vote",
    {
      "voter": "bola",
      "author": "petecorey",
      "permlink": "graphql-nosql-injection-through-json-types",
      "weight": 100
    }
  ]
}

{
  "trx_id": "cd867523d967b7262481b550beb84759e2b2b189",
  "block": 12821149,
  "trx_in_block": 2,
  "op_in_trx": 0,
  "virtual_op": 0,
  "timestamp": "2017-06-14T19:00:15",
  "op": [
    "vote",
    {
      "voter": "petecorey",
      "author": "allanmacgregor",
      "permlink": "you-should-learn-functional-programming-in-2017",
      "weight": 10000
    }
  ]
}