Jekyll2021-06-06T19:31:33+00:00http://www.cjadkins.com/feed.xmlCJAdkinsA little bit of math, a little bit of code and a game here and there. Let's learn things together.
Christopher J. AdkinsWorking Behind A Corporate Firewall With Open Source2020-03-06T00:00:00+00:002020-03-06T00:00:00+00:00http://www.cjadkins.com/docker/2020/03/06/working-behind-a-corporate-firewall<p>When I first started working with open source most the tools just worked! Life was wonderful on my personal machine and various package managers were easy to use. Then the big bad corporate firewall came into play and I had no idea what I was doing anymore, tools that used to work we’re encountering SSL/TLS errors and connections weren’t happening.</p>
<p>If you already know what you’re doing… this will be obvious to you. If you’re struggling with setting up environments behind your firewall, hopefully you find something useful here.</p>
<p>I’ll go over a few modifications to base Docker images that allow connections to private (corporate) and public package repositories for Python, Node, .NET and Debian.</p>
<p><a href="https://github.com/Softyy/docker-images-with-proxies">Link to repo files here</a></p>
<h2 id="python-pip">Python (pip)</h2>
<p>The main package manager folks use here is pip (conda is similar). So we’ll need a <code class="language-plaintext highlighter-rouge">pip.conf</code> file on Linux, or a <code class="language-plaintext highlighter-rouge">pip.ini</code> file on Windows. <a href="https://pip.pypa.io/en/stable/user_guide/#config-file">For full reference click this link</a>. For Debian, it’s nice to the proxy configs to <code class="language-plaintext highlighter-rouge">apt</code> as well with <code class="language-plaintext highlighter-rouge">apt.conf</code>. If you want to add a private repo for <code class="language-plaintext highlighter-rouge">apt</code>, I’d probably use <code class="language-plaintext highlighter-rouge">add-apt-repository</code> instead of adding the settings manually. Ok, so together we create these two files</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># pip.conf</span>
<span class="o">[</span>global]
<span class="c"># the default index-url is https://pypi.python.org/simple</span>
<span class="c"># fill in the blanks below to the location of your private repo (maybe you need to authenticate with it, [*] means optional)</span>
index-url <span class="o">=</span> https://[<username>:<password>@]<index_host>:<index_port>/<route_to_repo_index>
<span class="c"># if you need a proxy to access the index-url, you'll need to pass the proxy server details here</span>
<span class="c"># this is equivalent to pip install --proxy http://[<username>:<password>@]<proxy_host>:<proxy_port> <package_name></span>
proxy <span class="o">=</span> http://[<username>:<password>@]<proxy_host>:<proxy_port>
<span class="c"># also, if you're mucking around with some self-signed certs, don't forget to trust that host for good measure.</span>
trusted-host <span class="o">=</span> <host>
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># apt.conf</span>
Acquire::http::proxy <span class="s2">"http://[<username>:<password>@]<proxy_host>:<proxy_port>"</span>
Acquire::https::proxy <span class="s2">"http://[<username>:<password>@]<proxy_host>:<proxy_port>"</span>
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># .curlrc</span>
proxy <span class="o">=</span> http://[<username>:<password>@]<proxy_host>:<proxy_port>
</code></pre></div></div>
<p>With this two files, we can create a python base Dockerfile so any internal info to server references can be left outside future Dockerfiles. i.e, if we host the following docker image internally</p>
<div class="language-Dockerfile highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">ARG</span><span class="s"> PYTHON_VERSION=3.7</span>
<span class="k">FROM</span><span class="s"> python:${PYTHON_VERSION}</span>
<span class="k">LABEL</span><span class="s"> python_version=${PYTHON_VERSION}</span>
<span class="k">COPY</span><span class="s"> pip.conf /etc/pip.conf</span>
<span class="k">COPY</span><span class="s"> apt.conf /etc/apt/apt.conf</span>
<span class="k">COPY</span><span class="s"> .curlrc /root/.curlrc</span>
<span class="k">CMD</span><span class="s"> ["bash"]</span>
</code></pre></div></div>
<p>I can build an app internally with docker, but also build it externally if something changes down the road and we open source the image or something.</p>
<div class="language-Dockerfile highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">ARG</span><span class="s"> PYTHON_VERSION=3.7</span>
<span class="c">#FROM python:${PYTHON_VERSION}</span>
<span class="k">FROM</span><span class="s"> <python_internal>:${PYTHON_VERSION}</span>
<span class="k">RUN </span>apt-get update
<span class="k">RUN </span>curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
<span class="k">RUN </span>pip <span class="nb">install </span>pandas
...
</code></pre></div></div>
<p>Don’t forget to be mindful of if you’re installing from internal <code class="language-plaintext highlighter-rouge">apt</code> repo’s or <code class="language-plaintext highlighter-rouge">curl</code>ing from an internal host. Then you’re not really doing a fully open source solution internally, it requires slightly more setup at the repo file level.</p>
<h2 id="node-npm">Node (npm)</h2>
<p>Now-a-days most tools follow similar patterns, so all we’ll have to change for Node is the npm configs! Create a <code class="language-plaintext highlighter-rouge">.npmrc</code> file.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># .npmrc</span>
<span class="c"># the default registry is https://registry.npmjs.org/</span>
<span class="c"># fill in the blanks below to the location of your private repo (maybe you need to authenticate with it, [*] means optional)</span>
<span class="nv">registry</span><span class="o">=</span>https://[<username>:<password>@]<index_host>:<index_port>/<route_to_repo_index>
<span class="c"># if you need a proxy to access the index-url, you'll need to pass the proxy server details here</span>
<span class="c"># this is equivalent to npm install --proxy http://[<username>:<password>@]<proxy_host>:<proxy_port> <package_name></span>
<span class="nv">proxy</span><span class="o">=</span>http://[<username>:<password>@]<proxy_host>:<proxy_port>
https-proxy<span class="o">=</span>http://[<username>:<password>@]<proxy_host>:<proxy_port>
<span class="c"># also, if you're mucking around with some self-signed certs, you'll have to ignore them or...</span>
strict-ssl<span class="o">=</span><span class="nb">false</span>
<span class="c"># you can add the certs of all the hosts you'd like to trust like follows</span>
<span class="c"># ca[]="cert 1 base64 string"</span>
<span class="c"># ca[]="cert 2 base64 string"</span>
</code></pre></div></div>
<p>Equipped with our configs, we can create a node base image by following the same logic as before</p>
<div class="language-Dockerfile highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">ARG</span><span class="s"> NODE_VERSION=13.10</span>
<span class="k">FROM</span><span class="s"> node:${NODE_VERSION}</span>
<span class="k">LABEL</span><span class="s"> node_version=${NODE_VERSION}</span>
<span class="k">COPY</span><span class="s"> .npmrc /root/.npmrc</span>
<span class="k">COPY</span><span class="s"> apt.conf /etc/apt/apt.conf</span>
<span class="k">COPY</span><span class="s"> .curlrc /root/.curlrc</span>
<span class="k">CMD</span><span class="s"> ["bash"]</span>
</code></pre></div></div>
<p>Now you can use all the commands without worrying about the proxy!</p>
<h2 id="net-nuget">.NET (NuGet)</h2>
<p>Even with Microsoft, it’s the same story with .NET Core! Just create <code class="language-plaintext highlighter-rouge">NuGet.Config</code> with the following:</p>
<pre><code class="language-XML"><?xml version="1.0" encoding="utf-8"?>
<configuration>
<config>
<add key="dependencyVersion" value="Highest" />
<add key="https_proxy" value="http://[<username>:<password>@]<proxy_host>:<proxy_port>" />
<add key="http_proxy" value="http://[<username>:<password>@]<proxy_host>:<proxy_port>" />
</config>
<packageSources>
<add key="NuGet official package source" value="https://api.nuget.org/v3/index.json" />
</packageSources>
<trustedSigners>
<author name="microsoft">
<certificate fingerprint="3F9001EA83C560D712C24CF213C3D312CB3BFF51EE89435D3430BD06B5D0EECE" hashAlgorithm="SHA256" allowUntrustedRoot="false" />
</author>
<repository name="nuget.org" serviceIndex="https://api.nuget.org/v3/index.json">
<certificate fingerprint="0E5F38F57DC1BCC806D8494F4F90FBCEDD988B46760709CBEEC6F4219AA6157D" hashAlgorithm="SHA256" allowUntrustedRoot="false" />
<owners>microsoft;aspnet;nuget</owners>
</repository>
</trustedSigners>
</configuration>
</code></pre>
<p>and then our Dockerfile follows the same format as python and node</p>
<div class="language-Dockerfile highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">ARG</span><span class="s"> DOTNET_CORE_VERSION=3.1</span>
<span class="k">FROM</span><span class="s"> mcr.microsoft.com/dotnet/core/sdk:${DOTNET_CORE_VERSION}</span>
<span class="k">LABEL</span><span class="s"> dotnet_core_version=${DOTNET_CORE_VERSION}</span>
<span class="k">COPY</span><span class="s"> NuGet.Config /root/.nuget/NuGet/</span>
<span class="k">COPY</span><span class="s"> apt.conf /etc/apt/apt.conf</span>
<span class="k">COPY</span><span class="s"> .curlrc /root/.curlrc</span>
<span class="k">CMD</span><span class="s"> ["bash"]</span>
</code></pre></div></div>
<p>Now you can <code class="language-plaintext highlighter-rouge">dotnet restore</code> away</p>
<h3 id="remarks">Remarks</h3>
<p>If you’ve made it this far, thanks for reading! Good luck with your hunt for knowledge, the corporate world is a political one! Stay Sane!</p>Christopher J. AdkinsWhen I first started working with open source most the tools just worked! Life was wonderful on my personal machine and various package managers were easy to use. Then the big bad corporate firewall came into play and I had no idea what I was doing anymore, tools that used to work we’re encountering SSL/TLS errors and connections weren’t happening.Open Source Analytics In A Beautiful Dashboard? - Part 32018-11-23T00:00:00+00:002018-11-23T00:00:00+00:00http://www.cjadkins.com/dash/2018/11/23/open-source-analytics-dashboards-part-three<p>Picking up where we left off in the <a href="/dash/2018/11/16/open-source-analytics-dashboards-part-two.html">last post</a>, we’re going to start with 2 things:</p>
<ul>
<li>Setting up our database/model with an ORM (Object-Relational Mapper, we’ll use SQLAlchemy).</li>
<li>Setting up our caching server (Redis) with dash.</li>
</ul>
<p>The caching server is only needed in situations where you’re pulling data that multiple uses might need. If you’re using this for an example of analytics dashboard for multiple users, the caching serer may suit your needs.</p>
<p>Let’s begin by installing flask_caching, redis, and flask_sqlalchemy. With those installed, let’s create our database connection. Add the following to our <code class="language-plaintext highlighter-rouge">__init__.py</code></p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="p">...</span>
<span class="kn">from</span> <span class="nn">flask_sqlalchemy</span> <span class="kn">import</span> <span class="n">SQLAlchemy</span>
<span class="p">...</span>
<span class="n">app</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">config</span><span class="p">[</span><span class="s">'SQLALCHEMY_DATABASE_URI'</span><span class="p">]</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">getenv</span><span class="p">(</span><span class="s">'CONNECTION_STRING'</span><span class="p">)</span>
<span class="n">app</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">config</span><span class="p">[</span><span class="s">'SQLALCHEMY_TRACK_MODIFICATIONS'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">False</span> <span class="c1"># silence the deprecation warning
</span><span class="n">db</span> <span class="o">=</span> <span class="n">SQLAlchemy</span><span class="p">(</span><span class="n">app</span><span class="p">.</span><span class="n">server</span><span class="p">)</span>
<span class="p">...</span></code></pre></figure>
<p>We’ll set the connection string in the environment variables. Now let’s define our data model. Since the data we’re working with a credit card transaction, I’ve called the fill with the model <code class="language-plaintext highlighter-rouge">transaction.py</code> and I’ve defined the model according to the data we already have.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">webapp</span> <span class="kn">import</span> <span class="n">db</span>
<span class="kn">from</span> <span class="nn">..consts</span> <span class="kn">import</span> <span class="n">TRANSACTION_TABLE_NAME</span>
<span class="k">class</span> <span class="nc">Transaction</span><span class="p">(</span><span class="n">db</span><span class="p">.</span><span class="n">Model</span><span class="p">):</span>
<span class="n">__tablename__</span> <span class="o">=</span> <span class="n">TRANSACTION_TABLE_NAME</span>
<span class="nb">id</span> <span class="o">=</span> <span class="n">db</span><span class="p">.</span><span class="n">Column</span><span class="p">(</span><span class="n">db</span><span class="p">.</span><span class="n">Integer</span><span class="p">,</span><span class="n">unique</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">nullable</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">primary_key</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">date</span> <span class="o">=</span> <span class="n">db</span><span class="p">.</span><span class="n">Column</span><span class="p">(</span><span class="n">db</span><span class="p">.</span><span class="n">Date</span><span class="p">,</span><span class="n">nullable</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="n">vendor</span> <span class="o">=</span> <span class="n">db</span><span class="p">.</span><span class="n">Column</span><span class="p">(</span><span class="n">db</span><span class="p">.</span><span class="n">String</span><span class="p">,</span><span class="n">nullable</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="n">charge</span> <span class="o">=</span> <span class="n">db</span><span class="p">.</span><span class="n">Column</span><span class="p">(</span><span class="n">db</span><span class="p">.</span><span class="n">Float</span><span class="p">)</span>
<span class="n">credit</span> <span class="o">=</span> <span class="n">db</span><span class="p">.</span><span class="n">Column</span><span class="p">(</span><span class="n">db</span><span class="p">.</span><span class="n">Float</span><span class="p">)</span>
<span class="n">total_balance</span> <span class="o">=</span> <span class="n">db</span><span class="p">.</span><span class="n">Column</span><span class="p">(</span><span class="n">db</span><span class="p">.</span><span class="n">Float</span><span class="p">)</span></code></pre></figure>
<p>Now if we call <code class="language-plaintext highlighter-rouge">db.create_all()</code> this will create our table with a Schema that fits our Transaction class. Currently my data is scattered into csv’s, so using the method <code class="language-plaintext highlighter-rouge">load_data()</code> from our data manager, I’ll use pandas’ built in <code class="language-plaintext highlighter-rouge">to_sql()</code> to toss that data into the db if it hasn’t already done so. I’ve added a file called <code class="language-plaintext highlighter-rouge">startup.py</code> to run after initialization (by importing it in <code class="language-plaintext highlighter-rouge">__init__.py</code>).</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">webapp</span> <span class="kn">import</span> <span class="n">db</span><span class="p">,</span><span class="n">dm</span>
<span class="kn">from</span> <span class="nn">.models.transaction</span> <span class="kn">import</span> <span class="n">Transaction</span>
<span class="c1"># creates the table schema's if they do not already exist.
</span><span class="n">db</span><span class="p">.</span><span class="n">create_all</span><span class="p">()</span>
<span class="c1">#populate the database if the data isn't there.
</span><span class="k">if</span> <span class="ow">not</span> <span class="n">Transaction</span><span class="p">.</span><span class="n">query</span><span class="p">.</span><span class="n">first</span><span class="p">():</span>
<span class="n">dm</span><span class="p">.</span><span class="n">load_data</span><span class="p">()</span>
<span class="n">dm</span><span class="p">.</span><span class="n">data</span><span class="p">.</span><span class="n">to_sql</span><span class="p">(</span> <span class="n">name</span><span class="o">=</span><span class="n">Transaction</span><span class="p">.</span><span class="n">__tablename__</span><span class="p">,</span>
<span class="n">con</span><span class="o">=</span><span class="n">db</span><span class="p">.</span><span class="n">engine</span><span class="p">,</span>
<span class="n">if_exists</span> <span class="o">=</span> <span class="s">'append'</span><span class="p">,</span>
<span class="n">index</span><span class="o">=</span><span class="bp">False</span>
<span class="p">)</span>
<span class="n">db</span><span class="p">.</span><span class="n">session</span><span class="p">.</span><span class="n">commit</span><span class="p">()</span></code></pre></figure>
<p>Now let’s add a method to extract the data from the database in our data manager.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">load_from_db</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span><span class="n">db</span><span class="p">):</span>
<span class="k">with</span> <span class="n">db</span><span class="p">.</span><span class="n">engine</span><span class="p">.</span><span class="n">connect</span><span class="p">()</span> <span class="k">as</span> <span class="n">conn</span><span class="p">,</span> <span class="n">conn</span><span class="p">.</span><span class="n">begin</span><span class="p">():</span>
<span class="k">return</span> <span class="n">pd</span><span class="p">.</span><span class="n">read_sql</span><span class="p">(</span><span class="n">TRANSACTION_TABLE_NAME</span><span class="p">,</span><span class="n">conn</span><span class="p">)</span></code></pre></figure>
<p>Now we can replace all references to self.data with this method. The finishing touch is in the docker files, in our local override I’ve added the connection string and the database setup.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>...
environment:
- FLASK_ENV=development # this is a env var we've added to the container.
- CONNECTION_STRING=postgresql://postgres:password@db:5432/postgres #postgresql:///<username>:<password>@<host>:<port>/<database>
- REDIS_URL=redis://cache:6379
command: ["python", "run.py"] # this is an override of the CMD in the Dockerfile
db:
restart: always
environment:
- POSTGRES_PASSWORD=password
</code></pre></div></div>
<p>We’ll also need to include the drivers to establish the postgres connection in our Dockerfile, so update the C binaries line to include those.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># adding the C binaries for pandas and postgres drivers
RUN apk --update add --no-cache g++ libpq postgresql-dev
</code></pre></div></div>
<p>To setup the caching server is just as simple. We initialize the cache reference in <code class="language-plaintext highlighter-rouge">__init__.py</code></p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="p">...</span>
<span class="n">db</span> <span class="o">=</span> <span class="n">SQLAlchemy</span><span class="p">(</span><span class="n">app</span><span class="p">.</span><span class="n">server</span><span class="p">)</span>
<span class="n">CACHE_CONFIG</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">'CACHE_TYPE'</span><span class="p">:</span> <span class="s">'redis'</span><span class="p">,</span>
<span class="s">'CACHE_REDIS_URL'</span><span class="p">:</span> <span class="n">os</span><span class="p">.</span><span class="n">getenv</span><span class="p">(</span><span class="s">'REDIS_URL'</span><span class="p">),</span>
<span class="c1"># should be equal to maximum number of users on the app at a single time
</span> <span class="c1"># higher numbers will store more data in the redis cache
</span> <span class="s">'CACHE_THRESHOLD'</span><span class="p">:</span> <span class="mi">200</span>
<span class="p">}</span>
<span class="n">cache</span> <span class="o">=</span> <span class="n">Cache</span><span class="p">()</span>
<span class="n">cache</span><span class="p">.</span><span class="n">init_app</span><span class="p">(</span><span class="n">app</span><span class="p">.</span><span class="n">server</span><span class="p">,</span> <span class="n">config</span><span class="o">=</span><span class="n">CACHE_CONFIG</span><span class="p">)</span>
<span class="p">...</span></code></pre></figure>
<p>with reference to the env var REDIS_URL (just as we did for the db). Now around any function we’d like to cache. Simply add the cached / memorize decorators. For example I’ve cached the <code class="language-plaintext highlighter-rouge">load_from_db()</code> method in data manager.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="p">...</span>
<span class="kn">from</span> <span class="nn">webapp</span> <span class="kn">import</span> <span class="n">cache</span><span class="p">,</span><span class="n">db</span>
<span class="p">...</span>
<span class="o">@</span><span class="n">cache</span><span class="p">.</span><span class="n">memoize</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">load_from_db</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">with</span> <span class="n">db</span><span class="p">.</span><span class="n">engine</span><span class="p">.</span><span class="n">connect</span><span class="p">()</span> <span class="k">as</span> <span class="n">conn</span><span class="p">,</span> <span class="n">conn</span><span class="p">.</span><span class="n">begin</span><span class="p">():</span>
<span class="k">return</span> <span class="n">pd</span><span class="p">.</span><span class="n">read_sql</span><span class="p">(</span><span class="n">TRANSACTION_TABLE_NAME</span><span class="p">,</span><span class="n">conn</span><span class="p">)</span>
<span class="p">...</span></code></pre></figure>
<p>At this point we’ve basically set everything up. We’ve got our data, we’ve got some simple visualizations, and we’ve got our cache.</p>Christopher J. AdkinsPicking up where we left off in the last post, we’re going to start with 2 things:Open Source Analytics In A Beautiful Dashboard? - Part 22018-11-16T00:00:00+00:002018-11-16T00:00:00+00:00http://www.cjadkins.com/dash/2018/11/16/open-source-analytics-dashboards-part-two<p>Picking up where we left off in the <a href="/dash/2018/11/13/open-source-analytics-dashboards.html">last post</a>, we’re going to do some housekeeping. Namely, I’m going to frame this in the context of a docker container and ditch the virtual environment… If you don’t care about dockerizing this application, that’s fine, just skip ahead a bit. This will make the postgres addition a cake walk in the future, and remove the dependencies of having python on your machine… but adding a docker requirement instead (can’t win all your battles). So Let’s add a few files to the root level, namely the docker files.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>|-- Dockerfile
|-- README.md
|-- docker-compose.override.yml
|-- docker-compose.prod.yml
|-- docker-compose.yml
|-- requirements.txt
|-- run.py
|-- venv
`-- webapp
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">Dockerfile</code> simply let’s us define how to build the dash app, so we just copy our instructions of how we installed it before. I’m going to include a production grade server as well, <a href="https://gunicorn.org/">green unicorn</a>, but you can use something else if you like. To generate the <code class="language-plaintext highlighter-rouge">requirements.txt</code>, you can use <code class="language-plaintext highlighter-rouge">pip freeze > requirements.txt</code> from your old virtual env (or just use the one I’ve provided in the repo). With the requirements in hand, our <code class="language-plaintext highlighter-rouge">Dockerfile</code> should look something like</p>
<figure class="highlight"><pre><code class="language-docker" data-lang="docker"><span class="k">FROM</span><span class="s"> python:alpine3.7</span>
<span class="k">LABEL</span><span class="s"> version="1.0" description="a simple personal finance dashboard" maintainer="adkincj@gmail.com"</span>
<span class="c"># adding the C binaries for pandas </span>
<span class="k">RUN </span>apk <span class="nt">--update</span> add <span class="nt">--no-cache</span> g++
<span class="c"># creating the directory for our code</span>
<span class="k">RUN </span><span class="nb">mkdir</span> <span class="nt">-p</span> /app
<span class="k">COPY</span><span class="s"> . /app</span>
<span class="k">WORKDIR</span><span class="s"> /app</span>
<span class="c"># install our requirements for the project with the green unicorn server.</span>
<span class="k">RUN </span>pip <span class="nb">install</span> <span class="nt">-r</span> requirements.txt gunicorn
<span class="c"># open a port on the container so we can communicate with the app outside of the container.</span>
<span class="k">EXPOSE</span><span class="s"> 8050</span>
<span class="c"># start the dash app with green unicorn (and 2 workers).</span>
<span class="k">CMD</span><span class="s"> ["gunicorn","-w","2","--bind",":8050","webapp:app.server"]</span></code></pre></figure>
<p>I won’t go into too much detail about how the above works, just accept it does for now. Now let’s add the compose files in. The base <code class="language-plaintext highlighter-rouge">docker-compose.yml</code> has the template for our application. It looks something like</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">version</span><span class="pi">:</span> <span class="s1">'</span><span class="s">3'</span>
<span class="na">services</span><span class="pi">:</span>
<span class="na">dash</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">cjadkins/dash-finance-app</span>
<span class="na">links</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">db</span>
<span class="pi">-</span> <span class="s">cache</span>
<span class="na">db</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">postgres:latest</span>
<span class="na">cache</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">redis:latest</span></code></pre></figure>
<p>which you can see has a reference to the name of our dash image (which we will build), postgres (our relational db we’ll be using) and our caching server redis. For local development (what we’re doing right now) I’ve included an override to add in our local dev settings. So in <code class="language-plaintext highlighter-rouge">docker-compose.override.yml</code> we see our local settings</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">services</span><span class="pi">:</span>
<span class="na">dash</span><span class="pi">:</span>
<span class="na">build</span><span class="pi">:</span> <span class="s">.</span> <span class="c1"># runs the Dockerfile located in the current directory.</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">.:/app</span> <span class="c1"># this binds the current directory to the folder /app in the container</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">8050:8050</span> <span class="c1"># this maps port 8050 on the container to port 8050 on our host.</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">FLASK_ENV=development</span> <span class="c1"># this is a env var we've added to the container.</span>
<span class="na">command</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">python"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">run.py"</span><span class="pi">]</span> <span class="c1"># this is an override of the CMD in the Dockerfile</span></code></pre></figure>
<p>Here we’re overriding the command in the dockerfile, with the old <code class="language-plaintext highlighter-rouge">python run.py</code> we’ve been using to start the server and binding the code on our machine with the code in the container. This means that when we’re changing our code, the code in the container will change as well (even if it’s running). Now we can safely remove our <code class="language-plaintext highlighter-rouge">venv</code> and <code class="language-plaintext highlighter-rouge">.env</code> (but you can keep them around if you’re more comfortable with them, don’t forgot to add them to your <code class="language-plaintext highlighter-rouge">.dockerignore</code> if you keep them around), and start our local dev server with <code class="language-plaintext highlighter-rouge">docker-compose up</code> (docker-compose is bundled in docker for windows. If you’re on a Unix system. I’m fairly sure you need to get it separately). Last thing you’ll need to update is the <code class="language-plaintext highlighter-rouge">run.py</code> and update the bind:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">webapp</span> <span class="kn">import</span> <span class="n">app</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">app</span><span class="p">.</span><span class="n">run_server</span><span class="p">(</span><span class="n">debug</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span><span class="n">host</span><span class="o">=</span><span class="s">"0.0.0.0"</span><span class="p">,</span><span class="n">port</span><span class="o">=</span><span class="mi">8050</span><span class="p">)</span></code></pre></figure>
<p>Let’s get back to the app. The next short goal will implement is a selector for the vender, and get that change what our spending is for that vender. I’ve updated my load_data method to add all the my monthly csv files I got from the bank, since we’ll add the db connection piece later. I’ve also added a simple method to place the vender data into a format plotly likes(which follows <code class="language-plaintext highlighter-rouge">{'label':'This is shown as an option','value':'this is value of that option'}</code>):</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">load_data</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">files_path</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">os</span><span class="p">.</span><span class="n">getcwd</span><span class="p">(),</span><span class="s">"webapp"</span><span class="p">,</span><span class="s">"test_data"</span><span class="p">)</span>
<span class="n">all_files</span> <span class="o">=</span> <span class="n">glob</span><span class="p">.</span><span class="n">glob</span><span class="p">(</span><span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">files_path</span><span class="p">,</span> <span class="s">"*.csv"</span><span class="p">))</span>
<span class="bp">self</span><span class="p">.</span><span class="n">data</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="n">concat</span><span class="p">((</span><span class="n">pd</span><span class="p">.</span><span class="n">read_csv</span><span class="p">(</span><span class="n">f</span><span class="p">,</span><span class="n">names</span><span class="o">=</span><span class="n">DATA_COLUMNS</span><span class="p">)</span> <span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">all_files</span><span class="p">))</span>
<span class="bp">self</span><span class="p">.</span><span class="n">data</span><span class="p">[</span><span class="n">DATA_DATE</span><span class="p">]</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="n">to_datetime</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">data</span><span class="p">[</span><span class="n">DATA_DATE</span><span class="p">]).</span><span class="n">dt</span><span class="p">.</span><span class="n">date</span>
<span class="p">...</span>
<span class="k">def</span> <span class="nf">get_vendor_options</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">vendors</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">data</span><span class="p">[</span><span class="n">DATA_VENDOR</span><span class="p">].</span><span class="n">unique</span><span class="p">()</span>
<span class="n">options</span> <span class="o">=</span> <span class="p">[{</span><span class="s">'label'</span><span class="p">:</span><span class="s">'All'</span><span class="p">,</span><span class="s">'value'</span><span class="p">:</span><span class="mi">0</span><span class="p">}]</span>
<span class="n">options</span><span class="p">.</span><span class="n">extend</span><span class="p">([{</span><span class="s">'label'</span><span class="p">:</span><span class="n">vendors</span><span class="p">[</span><span class="n">n</span><span class="p">],</span><span class="s">'value'</span><span class="p">:</span><span class="n">n</span><span class="o">+</span><span class="mi">1</span><span class="p">}</span> <span class="k">for</span> <span class="n">n</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="nb">len</span><span class="p">(</span><span class="n">vendors</span><span class="p">))])</span>
<span class="k">return</span> <span class="n">options</span></code></pre></figure>
<p>This vendor method simply gives us all the selections that will work with the data loaded, so to add the options to dashboard we go to <code class="language-plaintext highlighter-rouge">index.py</code>.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">render</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">y</span><span class="p">,</span><span class="n">vendor_options</span><span class="o">=</span><span class="n">VENDOR_TEST_OPTIONS</span><span class="p">):</span>
<span class="k">return</span> <span class="n">html</span><span class="p">.</span><span class="n">Div</span><span class="p">(</span><span class="n">children</span><span class="o">=</span><span class="p">[</span>
<span class="p">...</span>
<span class="n">dcc</span><span class="p">.</span><span class="n">Dropdown</span><span class="p">(</span>
<span class="nb">id</span><span class="o">=</span><span class="n">VENDOR_SELECTOR_ID</span><span class="p">,</span>
<span class="n">options</span><span class="o">=</span><span class="n">vendor_options</span><span class="p">,</span>
<span class="n">multi</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
<span class="n">value</span><span class="o">=</span><span class="p">[</span><span class="n">vendor_options</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="s">'value'</span><span class="p">]]</span>
<span class="p">),</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">VENDOR_SELECTOR_ID</span> <span class="o">=</span> <span class="s">'vendor-selection'</span>
<span class="n">VENDOR_TEST_OPTIONS</span> <span class="o">=</span> <span class="p">[</span>
<span class="p">{</span><span class="s">'label'</span><span class="p">:</span> <span class="s">'All'</span><span class="p">,</span> <span class="s">'value'</span><span class="p">:</span> <span class="mi">0</span><span class="p">},</span>
<span class="p">{</span><span class="s">'label'</span><span class="p">:</span> <span class="s">'Vendor 1'</span><span class="p">,</span> <span class="s">'value'</span><span class="p">:</span> <span class="mi">1</span><span class="p">},</span>
<span class="p">{</span><span class="s">'label'</span><span class="p">:</span> <span class="s">'Vendor 2'</span><span class="p">,</span> <span class="s">'value'</span><span class="p">:</span> <span class="mi">2</span><span class="p">},</span>
<span class="p">{</span><span class="s">'label'</span><span class="p">:</span> <span class="s">'Vendor 3'</span><span class="p">,</span> <span class="s">'value'</span><span class="p">:</span> <span class="mi">3</span><span class="p">}</span>
<span class="p">]</span></code></pre></figure>
<p>This adds the options to the dashboard, but it currently doesn’t do anything to the page. So let’s add a callback for some action. First we’ll update our data manager with the method to filter for our selection. You’ll see that the values of the dropdown element will always come back as an <code class="language-plaintext highlighter-rouge">[int]</code>, so we’ll plan for an index match to see what’s included. Also, I realized we aren’t filling null dates at this point, and I think the graph looks better with the 0’s included. So Let’s make the following edits to <code class="language-plaintext highlighter-rouge">data.py</code></p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">get_charge_series_tuple</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span><span class="n">data</span><span class="o">=</span><span class="n">pd</span><span class="p">.</span><span class="n">DataFrame</span><span class="p">()):</span>
<span class="c1"># take the default argument to be the data frame within the manager.
</span> <span class="k">if</span> <span class="n">data</span><span class="p">.</span><span class="n">empty</span><span class="p">:</span>
<span class="n">data</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">data</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">series</span> <span class="o">=</span> <span class="n">data</span><span class="p">.</span><span class="n">groupby</span><span class="p">([</span><span class="n">DATA_DATE</span><span class="p">]).</span><span class="nb">sum</span><span class="p">()[</span><span class="n">DATA_CHARGE</span><span class="p">]</span>
<span class="n">index_range</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="n">date_range</span><span class="p">(</span><span class="nb">min</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">data</span><span class="p">[</span><span class="n">DATA_DATE</span><span class="p">]),</span><span class="nb">max</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">data</span><span class="p">[</span><span class="n">DATA_DATE</span><span class="p">]))</span>
<span class="n">series</span> <span class="o">=</span> <span class="n">series</span><span class="p">.</span><span class="n">reindex</span><span class="p">(</span><span class="n">index_range</span><span class="p">,</span><span class="n">fill_value</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="k">return</span> <span class="p">(</span><span class="n">series</span><span class="p">.</span><span class="n">index</span><span class="p">,</span><span class="n">series</span><span class="p">.</span><span class="n">values</span><span class="p">)</span>
<span class="k">except</span> <span class="nb">AttributeError</span><span class="p">:</span>
<span class="k">raise</span> <span class="nb">Exception</span><span class="p">(</span><span class="n">DATA_MANAGER_NO_DATA</span><span class="p">)</span>
<span class="k">except</span> <span class="nb">KeyError</span><span class="p">:</span>
<span class="k">raise</span> <span class="nb">Exception</span><span class="p">(</span><span class="n">DATA_MANAGER_KEY_ERROR</span><span class="p">)</span>
<span class="p">...</span>
<span class="k">def</span> <span class="nf">get_selected_charge_series_tuple</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span><span class="n">selection</span> <span class="o">=</span> <span class="p">[</span><span class="mi">0</span><span class="p">]):</span>
<span class="n">vendors</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">data</span><span class="p">[</span><span class="n">DATA_VENDOR</span><span class="p">].</span><span class="n">unique</span><span class="p">()</span>
<span class="n">selected_vendors</span> <span class="o">=</span> <span class="p">[</span><span class="n">element</span> <span class="k">for</span> <span class="n">index</span><span class="p">,</span> <span class="n">element</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">vendors</span><span class="p">)</span> <span class="k">if</span> <span class="n">index</span><span class="o">+</span><span class="mi">1</span> <span class="ow">in</span> <span class="n">selection</span><span class="p">]</span>
<span class="n">filtered_data</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">data</span><span class="p">[</span><span class="bp">self</span><span class="p">.</span><span class="n">data</span><span class="p">[</span><span class="n">DATA_VENDOR</span><span class="p">].</span><span class="n">isin</span><span class="p">(</span><span class="n">selected_vendors</span><span class="p">)]</span> <span class="k">if</span> <span class="ow">not</span> <span class="mi">0</span> <span class="ow">in</span> <span class="n">selection</span> <span class="k">else</span> <span class="bp">self</span><span class="p">.</span><span class="n">data</span>
<span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">get_charge_series_tuple</span><span class="p">(</span><span class="n">filtered_data</span><span class="p">)</span></code></pre></figure>
<p>Now our callback is simple to create, we just have to update the figure we’re showing. So let’s create a file called <code class="language-plaintext highlighter-rouge">callback.py</code> and include the one callback:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">dash.dependencies</span> <span class="kn">import</span> <span class="n">Input</span><span class="p">,</span><span class="n">Output</span>
<span class="kn">from</span> <span class="nn">webapp</span> <span class="kn">import</span> <span class="n">app</span><span class="p">,</span> <span class="n">dm</span>
<span class="kn">from</span> <span class="nn">..consts</span> <span class="kn">import</span> <span class="n">VENDOR_SELECTOR_ID</span><span class="p">,</span><span class="n">OUTPUT_GRAPH_ID</span>
<span class="kn">from</span> <span class="nn">webapp.managers.graph</span> <span class="kn">import</span> <span class="n">createFigure</span><span class="p">,</span><span class="n">createScatterTrace</span>
<span class="o">@</span><span class="n">app</span><span class="p">.</span><span class="n">callback</span><span class="p">(</span>
<span class="n">Output</span><span class="p">(</span><span class="n">OUTPUT_GRAPH_ID</span><span class="p">,</span><span class="s">'figure'</span><span class="p">),</span>
<span class="p">[</span><span class="n">Input</span><span class="p">(</span><span class="n">VENDOR_SELECTOR_ID</span><span class="p">,</span><span class="s">'value'</span><span class="p">)])</span>
<span class="k">def</span> <span class="nf">selector_change</span><span class="p">(</span><span class="n">selected_values</span><span class="p">):</span>
<span class="n">x</span><span class="p">,</span> <span class="n">y</span> <span class="o">=</span> <span class="n">dm</span><span class="p">.</span><span class="n">get_selected_charge_series_tuple</span><span class="p">(</span><span class="n">selected_values</span><span class="p">)</span>
<span class="k">return</span> <span class="n">createFigure</span><span class="p">([</span><span class="n">createScatterTrace</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">y</span><span class="p">)])</span></code></pre></figure>
<p>The syntax here might be a little weird for you, but we’re adding a callback to dash app (app) which will trigger when there’s a change to the property <code class="language-plaintext highlighter-rouge">value</code> of the HTML element <code class="language-plaintext highlighter-rouge">id=VENDOR_SELECTOR_ID</code>. The name of the function below the attribute isn’t important since the function directly below get’s called. In our case we’re updating the <code class="language-plaintext highlighter-rouge">figure</code> property of the HTML element with <code class="language-plaintext highlighter-rouge">id=OUTPUT_GRAPH_ID</code>. Lastly we just have to include this callback to the webapp by importing it in the <code class="language-plaintext highlighter-rouge">__init__.py</code> file.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="p">...</span>
<span class="c1"># load the data
</span><span class="n">dm</span> <span class="o">=</span> <span class="n">DataManager</span><span class="p">()</span>
<span class="n">dm</span><span class="p">.</span><span class="n">load_data</span><span class="p">()</span>
<span class="n">x</span><span class="p">,</span><span class="n">y</span> <span class="o">=</span> <span class="n">dm</span><span class="p">.</span><span class="n">get_charge_series_tuple</span><span class="p">()</span>
<span class="n">vendor_options</span> <span class="o">=</span> <span class="n">dm</span><span class="p">.</span><span class="n">get_vendor_options</span><span class="p">()</span>
<span class="n">app</span><span class="p">.</span><span class="n">layout</span> <span class="o">=</span> <span class="n">index</span><span class="p">.</span><span class="n">render</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">y</span><span class="p">,</span><span class="n">vendor_options</span><span class="p">)</span>
<span class="kn">from</span> <span class="nn">.callbacks</span> <span class="kn">import</span> <span class="n">callback</span></code></pre></figure>
<p>Now let’s have a look at what we have so far. My dashboard is looking something like:</p>
<p><img src="/assets/posts/dash-finance-app/second_template.gif" alt="Callbacks in action" /></p>
<p>The last thing we’ll add in this tutorial part is another selector for the date, so we may easily change the bounds of the graph. Using the re-indexing trick we used earlier this is a cake walk. We simply need to add the method arguments and the data will bound itself.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">get_charge_series_tuple</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span><span class="n">data</span><span class="o">=</span><span class="n">pd</span><span class="p">.</span><span class="n">DataFrame</span><span class="p">(),</span><span class="n">min_date</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span><span class="n">max_date</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
<span class="p">...</span>
<span class="k">if</span> <span class="n">min_date</span> <span class="o">==</span> <span class="bp">None</span><span class="p">:</span>
<span class="n">min_date</span> <span class="o">=</span> <span class="nb">min</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">data</span><span class="p">[</span><span class="n">DATA_DATE</span><span class="p">])</span>
<span class="k">if</span> <span class="n">max_date</span> <span class="o">==</span> <span class="bp">None</span><span class="p">:</span>
<span class="n">max_date</span> <span class="o">=</span> <span class="nb">max</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">data</span><span class="p">[</span><span class="n">DATA_DATE</span><span class="p">])</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">series</span> <span class="o">=</span> <span class="n">data</span><span class="p">.</span><span class="n">groupby</span><span class="p">([</span><span class="n">DATA_DATE</span><span class="p">]).</span><span class="nb">sum</span><span class="p">()[</span><span class="n">DATA_CHARGE</span><span class="p">]</span>
<span class="n">index_range</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="n">date_range</span><span class="p">(</span><span class="n">min_date</span><span class="p">,</span><span class="n">max_date</span><span class="p">)</span>
<span class="p">...</span>
<span class="k">def</span> <span class="nf">get_selected_charge_series_tuple</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span><span class="n">selection</span> <span class="o">=</span> <span class="p">[</span><span class="mi">0</span><span class="p">],</span><span class="n">min_date</span><span class="o">=</span><span class="n">MIN_DATE_SELECTOR</span><span class="p">,</span><span class="n">max_date</span><span class="o">=</span><span class="n">MAX_DATE_SELECTOR</span><span class="p">):</span>
<span class="p">...</span>
<span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">get_charge_series_tuple</span><span class="p">(</span><span class="n">filtered_data</span><span class="p">,</span><span class="n">min_date</span><span class="p">,</span><span class="n">max_date</span><span class="p">)</span></code></pre></figure>
<p>Next up let’s throw the HTML in with the handy dash component into <code class="language-plaintext highlighter-rouge">index.py</code>.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="p">...</span>
<span class="n">dcc</span><span class="p">.</span><span class="n">DatePickerRange</span><span class="p">(</span>
<span class="nb">id</span><span class="o">=</span><span class="n">DATE_SELECTOR_ID</span><span class="p">,</span>
<span class="n">minimum_nights</span><span class="o">=</span><span class="mi">7</span><span class="p">,</span>
<span class="n">clearable</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
<span class="n">with_portal</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
<span class="n">min_date_allowed</span><span class="o">=</span><span class="n">MIN_DATE_SELECTOR</span><span class="p">,</span>
<span class="n">start_date</span><span class="o">=</span><span class="n">MIN_DATE_SELECTOR</span><span class="p">,</span>
<span class="n">max_date_allowed</span><span class="o">=</span><span class="n">MAX_DATE_SELECTOR</span><span class="p">,</span>
<span class="n">end_date</span><span class="o">=</span><span class="n">MAX_DATE_SELECTOR</span>
<span class="p">),</span>
<span class="p">...</span></code></pre></figure>
<p>Now we just have to add the callback. Since we’ve updated the method already, we just have to add the inputs to the callback.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">@</span><span class="n">app</span><span class="p">.</span><span class="n">callback</span><span class="p">(</span>
<span class="n">Output</span><span class="p">(</span><span class="n">OUTPUT_GRAPH_ID</span><span class="p">,</span><span class="s">'figure'</span><span class="p">),</span>
<span class="p">[</span><span class="n">Input</span><span class="p">(</span><span class="n">VENDOR_SELECTOR_ID</span><span class="p">,</span><span class="s">'value'</span><span class="p">),</span>
<span class="n">Input</span><span class="p">(</span><span class="n">DATE_SELECTOR_ID</span><span class="p">,</span> <span class="s">'start_date'</span><span class="p">),</span><span class="n">Input</span><span class="p">(</span><span class="n">DATE_SELECTOR_ID</span><span class="p">,</span> <span class="s">'end_date'</span><span class="p">)])</span>
<span class="k">def</span> <span class="nf">selector_change</span><span class="p">(</span><span class="n">selected_values</span><span class="p">,</span><span class="n">min_date</span><span class="p">,</span><span class="n">max_date</span><span class="p">):</span>
<span class="n">x</span><span class="p">,</span> <span class="n">y</span> <span class="o">=</span> <span class="n">dm</span><span class="p">.</span><span class="n">get_selected_charge_series_tuple</span><span class="p">(</span><span class="n">selected_values</span><span class="p">,</span><span class="n">min_date</span><span class="p">,</span><span class="n">max_date</span><span class="p">)</span>
<span class="k">return</span> <span class="n">createFigure</span><span class="p">([</span><span class="n">createScatterTrace</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">y</span><span class="p">)])</span></code></pre></figure>
<p>That ends of this part of the tutorial, next time we’ll add more views of the data and give ourselves the ability to drill down.</p>
<p><img src="/assets/posts/dash-finance-app/third_template.gif" alt="The end product of this part" /></p>Christopher J. AdkinsPicking up where we left off in the last post, we’re going to do some housekeeping. Namely, I’m going to frame this in the context of a docker container and ditch the virtual environment… If you don’t care about dockerizing this application, that’s fine, just skip ahead a bit. This will make the postgres addition a cake walk in the future, and remove the dependencies of having python on your machine… but adding a docker requirement instead (can’t win all your battles). So Let’s add a few files to the root level, namely the docker files.Open Source Analytics In A Beautiful Dashboard? - Part 12018-11-13T00:00:00+00:002018-11-13T00:00:00+00:00http://www.cjadkins.com/dash/2018/11/13/open-source-analytics-dashboards<h2 id="the-story">The Story</h2>
<p>Working for large Canadian telecommunications company has introduced me to the complexities and costs of how Business Intelligence (BI) groups operate. Usually it’s along the lines of some MicroStrategy / Tableau solution with restricted / expensive licenses. Not great since their software scales with money, not skill (I guess a little bit of skill :S). I decided to take it on myself to change how things were done, and modernize the team with some more advanced tooling, mostly Python / Jupyter but that got the ball rolling. Soon after I discovered <a href="https://plot.ly/products/dash/">Dash by Plotly</a> and it seemed like the perfect solution for providing some of our business analysts (BAs) the tools to refresh their dashboards. So we’ll go over how to create a simple dash app, namely one to give you some insights into your spending habits with database and all. I’ll be hand holding a bit, so feel free to skip ahead and <a href="https://github.com/Softyy/dash-personal-finance-example">check out the repo</a> to do with it what you will.</p>
<p><img src="/assets/posts/dash-finance-app/first_template.gif" alt="The end product of this part" /></p>
<h2 id="the-marriage-of-flask--react--plotlyjs">The Marriage of Flask + React + Plotly.js</h2>
<p>Dash is simple web framework with flask in the back, react in the front, and plotly.js for all the pretty pictures. The benefit is it’s all done in Python… well… you can’t really escape HTML or CSS but that’s beside the point. FYI I’m assuming you have Python 3+ and we’ll start from the Hello World example in plotly’s docs. First let’s install the packages we need.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip install dash dash_core_components dash_html_components python-dotenv
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">dotenv</code> package is the lazy mans way to set environment variables when you come to the project. So I’ve also created a <code class="language-plaintext highlighter-rouge">.env</code> file with the one line</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>FLASK_ENV=development
</code></pre></div></div>
<p>Here’s the layout I like to use for dash apps:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>|-- .env
|-- requirements.txt
|-- run.py
`-- webapp
|-- __init__.py
`-- templates
`-- index.py
</code></pre></div></div>
<p>where <code class="language-plaintext highlighter-rouge">webapp/templates</code> holds my “HTML” code. Here’s what <code class="language-plaintext highlighter-rouge">index.py</code> looks like</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">import</span> <span class="nn">dash_core_components</span> <span class="k">as</span> <span class="n">dcc</span>
<span class="kn">import</span> <span class="nn">dash_html_components</span> <span class="k">as</span> <span class="n">html</span>
<span class="k">def</span> <span class="nf">render</span><span class="p">():</span>
<span class="k">return</span> <span class="n">html</span><span class="p">.</span><span class="n">Div</span><span class="p">(</span><span class="n">children</span><span class="o">=</span><span class="p">[</span>
<span class="n">html</span><span class="p">.</span><span class="n">H1</span><span class="p">(</span><span class="n">children</span><span class="o">=</span><span class="s">'Hello Dash'</span><span class="p">),</span>
<span class="n">html</span><span class="p">.</span><span class="n">Div</span><span class="p">(</span><span class="n">children</span><span class="o">=</span><span class="s">'''
Dash: A web application framework for Python.
'''</span><span class="p">),</span>
<span class="n">dcc</span><span class="p">.</span><span class="n">Graph</span><span class="p">(</span>
<span class="nb">id</span><span class="o">=</span><span class="s">'example-graph'</span><span class="p">,</span>
<span class="n">figure</span><span class="o">=</span><span class="p">{</span>
<span class="s">'data'</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span><span class="s">'x'</span><span class="p">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">],</span> <span class="s">'y'</span><span class="p">:</span> <span class="p">[</span><span class="mi">4</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">],</span> <span class="s">'type'</span><span class="p">:</span> <span class="s">'bar'</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="s">'SF'</span><span class="p">},</span>
<span class="p">{</span><span class="s">'x'</span><span class="p">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">],</span> <span class="s">'y'</span><span class="p">:</span> <span class="p">[</span><span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">],</span> <span class="s">'type'</span><span class="p">:</span> <span class="s">'bar'</span><span class="p">,</span> <span class="s">'name'</span><span class="p">:</span> <span class="sa">u</span><span class="s">'Montréal'</span><span class="p">},</span>
<span class="p">],</span>
<span class="s">'layout'</span><span class="p">:</span> <span class="p">{</span>
<span class="s">'title'</span><span class="p">:</span> <span class="s">'Dash Data Visualization'</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">)</span>
<span class="p">])</span></code></pre></figure>
<p>It’s nice to wrap your html snippets in a function, so you can inject in whatever on generation. The rest of the server is just in the <code class="language-plaintext highlighter-rouge">__init__.py</code> file.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">import</span> <span class="nn">dash</span>
<span class="kn">from</span> <span class="nn">.templates</span> <span class="kn">import</span> <span class="n">index</span>
<span class="n">external_stylesheets</span> <span class="o">=</span> <span class="p">[</span><span class="s">'https://codepen.io/chriddyp/pen/bWLwgP.css'</span><span class="p">]</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">dash</span><span class="p">.</span><span class="n">Dash</span><span class="p">(</span><span class="n">__name__</span><span class="p">,</span> <span class="n">external_stylesheets</span><span class="o">=</span><span class="n">external_stylesheets</span><span class="p">)</span>
<span class="n">app</span><span class="p">.</span><span class="n">layout</span> <span class="o">=</span> <span class="n">index</span><span class="p">.</span><span class="n">render</span><span class="p">()</span></code></pre></figure>
<p>The last bit is <code class="language-plaintext highlighter-rouge">run.py</code>, which is the script I call to start the flask server for local dev work. You should use something like green unicorn for a more serious server.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">webapp</span> <span class="kn">import</span> <span class="n">app</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">app</span><span class="p">.</span><span class="n">run_server</span><span class="p">(</span><span class="n">debug</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span></code></pre></figure>
<p>You can now easily start the server with <code class="language-plaintext highlighter-rouge">python run.py</code>, and we’ve got a nice Hello World layout to start with. I’ve pulled some data from my credit card (in form of a .csv), we’ll load this into a postgres database soon, but let’s have a look at the structure my bank gives the data ( Your bank / credit provider may have a slightly different format, so take note!)</p>
<table>
<thead>
<tr>
<th>Date</th>
<th>Vendor</th>
<th>Charge</th>
<th>Credit</th>
<th>Total Balance</th>
</tr>
</thead>
<tbody>
<tr>
<td>01/10/2018</td>
<td>SKYWAY CIGAR</td>
<td>9.73</td>
<td> </td>
<td>68.07</td>
</tr>
<tr>
<td>01/11/2018</td>
<td>MCDONALD’S #40548</td>
<td>1.84</td>
<td> </td>
<td>69.91</td>
</tr>
<tr>
<td>01/11/2018</td>
<td>MCDONALD’S #40548</td>
<td>2.05</td>
<td> </td>
<td>71.96</td>
</tr>
<tr>
<td>01/12/2018</td>
<td>PLAYSTATION NETWORK</td>
<td>66.58</td>
<td> </td>
<td>162.04</td>
</tr>
<tr>
<td>01/12/2018</td>
<td>CROCODILE ROCK</td>
<td>23.5</td>
<td> </td>
<td>95.46</td>
</tr>
<tr>
<td>01/13/2018</td>
<td>LCBO/RAO #0511</td>
<td>36.9</td>
<td> </td>
<td>198.94</td>
</tr>
<tr>
<td>01/13/2018</td>
<td>UBER TRIP HQQFO HELP.UBER</td>
<td>15.49</td>
<td> </td>
<td>214.43</td>
</tr>
<tr>
<td>01/14/2018</td>
<td>PIZZA PIZZA LTD</td>
<td>34</td>
<td> </td>
<td>259.7</td>
</tr>
<tr>
<td>01/14/2018</td>
<td>UBER TRIP MBMCU HELP.UBER</td>
<td>11.27</td>
<td> </td>
<td>225.7</td>
</tr>
<tr>
<td>01/15/2018</td>
<td>TIM HORTONS 2115 QTH</td>
<td>5.9</td>
<td> </td>
<td>265.6</td>
</tr>
<tr>
<td>01/15/2018</td>
<td>HERO CERTIFIED BURGERS</td>
<td>6.76</td>
<td> </td>
<td>272.36</td>
</tr>
<tr>
<td>01/16/2018</td>
<td>PRESTO</td>
<td>20</td>
<td> </td>
<td>297.56</td>
</tr>
</tbody>
</table>
<p>I always like to get something up and running quickly, then iterate on top of that. Let’s focus on the Charge column and get us a nice time series of our charge history. My data happened to be unlabeled, so I have a file called <code class="language-plaintext highlighter-rouge">consts.py</code> with the labels I’d like to work with:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">DATA_COLUMNS</span> <span class="o">=</span> <span class="p">[</span><span class="s">'Date'</span><span class="p">,</span><span class="s">'Vendor'</span><span class="p">,</span><span class="s">'Charge'</span><span class="p">,</span><span class="s">'Credit'</span><span class="p">,</span><span class="s">'Total Balance'</span><span class="p">]</span>
<span class="n">DATA_DATE</span> <span class="o">=</span> <span class="s">'Date'</span>
<span class="n">DATA_CHARGE</span> <span class="o">=</span> <span class="s">'Charge'</span></code></pre></figure>
<p>Then I’ve created a folder called <code class="language-plaintext highlighter-rouge">managers</code> in webapp, and I’ll put in a simple object to input my data into and spit out the time series. In <code class="language-plaintext highlighter-rouge">data.py</code> we’ll create the DataManager:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="n">pd</span>
<span class="kn">from</span> <span class="nn">..consts</span> <span class="kn">import</span> <span class="n">DATA_COLUMNS</span><span class="p">,</span><span class="n">DATA_DATE</span><span class="p">,</span><span class="n">DATA_CHARGE</span>
<span class="k">class</span> <span class="nc">DataManager</span><span class="p">():</span>
<span class="k">def</span> <span class="nf">load_csv_data</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span><span class="n">filename</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">data</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="n">read_csv</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span><span class="n">names</span><span class="o">=</span><span class="n">DATA_COLUMNS</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">load_test_data</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">load_csv_data</span><span class="p">(</span><span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">os</span><span class="p">.</span><span class="n">getcwd</span><span class="p">(),</span><span class="s">"webapp"</span><span class="p">,</span><span class="s">"test_data"</span><span class="p">,</span><span class="s">"Jan-2018.csv"</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">get_charge_series_tuple</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">"""
Separates the date and charge values for easy plotting from the loaded data.
returns (date_array,charge_array)
"""</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">series</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">data</span><span class="p">.</span><span class="n">groupby</span><span class="p">([</span><span class="n">DATA_DATE</span><span class="p">]).</span><span class="nb">sum</span><span class="p">()[</span><span class="n">DATA_CHARGE</span><span class="p">]</span>
<span class="k">return</span> <span class="p">(</span><span class="n">series</span><span class="p">.</span><span class="n">index</span><span class="p">.</span><span class="n">values</span><span class="p">,</span><span class="n">series</span><span class="p">.</span><span class="n">values</span><span class="p">)</span>
<span class="k">except</span> <span class="nb">AttributeError</span><span class="p">:</span>
<span class="k">raise</span> <span class="nb">Exception</span><span class="p">(</span><span class="s">'Data must be loaded into DataManager'</span><span class="p">)</span>
<span class="k">except</span> <span class="nb">KeyError</span><span class="p">:</span>
<span class="k">raise</span> <span class="nb">Exception</span><span class="p">(</span><span class="s">'Column names of your data do not match your keys'</span><span class="p">)</span></code></pre></figure>
<p>Now that we have a simple way to interact with the data, let’s throw it into the plotly.js chart. As you probably noticed, I separate everything out since I prefer things small and reusable. So I’ve added another file to our managers folder, which I’ve called <code class="language-plaintext highlighter-rouge">graph.py</code>, which basically will contain our simple layer to plotly’s graph objects. Since all we need is a scatter plot, all we need is</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">import</span> <span class="nn">plotly.graph_objs</span> <span class="k">as</span> <span class="n">go</span>
<span class="k">def</span> <span class="nf">createScatterTrace</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">mode</span><span class="o">=</span><span class="s">'lines+markers'</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s">'unnamed-trace'</span><span class="p">):</span>
<span class="k">return</span> <span class="n">go</span><span class="p">.</span><span class="n">Scatter</span><span class="p">(</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">x</span><span class="p">,</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">y</span><span class="p">,</span>
<span class="n">mode</span> <span class="o">=</span> <span class="n">mode</span><span class="p">,</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
<span class="p">)</span></code></pre></figure>
<p>Now we just have to change out the trace in our rendered index page. I’ve modified some of the strings laying around in their and moved them to <code class="language-plaintext highlighter-rouge">consts.py</code>. Your index should look something like this</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="p">...</span>
<span class="kn">from</span> <span class="nn">..managers.graph</span> <span class="kn">import</span> <span class="n">createScatterTrace</span>
<span class="kn">from</span> <span class="nn">..consts</span> <span class="kn">import</span> <span class="n">WEBAPP_TITLE</span><span class="p">,</span> <span class="n">WEBAPP_SUBTITLE</span><span class="p">,</span> <span class="n">WEBAPP_GRAPH_TITLE</span>
<span class="k">def</span> <span class="nf">render</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">y</span><span class="p">):</span>
<span class="k">return</span> <span class="n">html</span><span class="p">.</span><span class="n">Div</span><span class="p">(</span><span class="n">children</span><span class="o">=</span><span class="p">[</span>
<span class="n">html</span><span class="p">.</span><span class="n">H1</span><span class="p">(</span><span class="n">children</span><span class="o">=</span><span class="n">WEBAPP_TITLE</span><span class="p">),</span>
<span class="n">html</span><span class="p">.</span><span class="n">H5</span><span class="p">(</span><span class="n">children</span><span class="o">=</span><span class="n">WEBAPP_SUBTITLE</span><span class="p">),</span>
<span class="n">dcc</span><span class="p">.</span><span class="n">Graph</span><span class="p">(</span>
<span class="nb">id</span><span class="o">=</span><span class="s">'example-graph'</span><span class="p">,</span>
<span class="n">figure</span><span class="o">=</span><span class="p">{</span>
<span class="s">'data'</span><span class="p">:</span> <span class="p">[</span>
<span class="n">createScatterTrace</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">y</span><span class="p">)</span>
<span class="p">],</span>
<span class="s">'layout'</span><span class="p">:</span> <span class="p">{</span>
<span class="s">'title'</span><span class="p">:</span> <span class="n">WEBAPP_GRAPH_TITLE</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">)</span>
<span class="p">])</span></code></pre></figure>
<p>Notice that we’re passing the data as a parameter of render, which means we’ll need to initialize our data manager and pass the data into our layout before we can assign it to app.layout. The last change is now to our apps <code class="language-plaintext highlighter-rouge">__init__.py</code> file, we simply call on DataManager:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="p">...</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">dash</span><span class="p">.</span><span class="n">Dash</span><span class="p">(</span><span class="n">__name__</span><span class="p">,</span> <span class="n">external_stylesheets</span><span class="o">=</span><span class="n">external_stylesheets</span><span class="p">)</span>
<span class="c1"># load the data
</span><span class="n">dm</span> <span class="o">=</span> <span class="n">DataManager</span><span class="p">()</span>
<span class="n">dm</span><span class="p">.</span><span class="n">load_test_data</span><span class="p">()</span>
<span class="n">x</span><span class="p">,</span><span class="n">y</span> <span class="o">=</span> <span class="n">dm</span><span class="p">.</span><span class="n">get_charge_series_tuple</span><span class="p">()</span>
<span class="n">app</span><span class="p">.</span><span class="n">layout</span> <span class="o">=</span> <span class="n">index</span><span class="p">.</span><span class="n">render</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">y</span><span class="p">)</span></code></pre></figure>
<p>Now we have a basic implementation of our data in a dashboard! The next part will deal with more data manipulation with callback’s so we can visualize the data better.</p>Christopher J. AdkinsThe StoryDockerizing a static web server2018-10-30T00:00:00+00:002018-10-30T00:00:00+00:00http://www.cjadkins.com/docker/2018/10/30/serving-a-static-web-page-docker<p>These days I’ve been dockerizing everything, and love the fact that docker allows for a very transparent deployment process with the server management side of things. Currently I’m working for a Canadian telecommunications company, and updating some of their web development processes and pipelines using the power of docker. There is something satisfying about typing <code class="language-plaintext highlighter-rouge">docker-compose up -d</code> and that’s it (well, I guess I’ve set some of the older guys with a <a href="https://www.portainer.io/">Portainer</a> GUI, semantics!)</p>
<p>I’ve created a repo with all the files <a href="https://github.com/Softyy/static-web-docker">here</a>, so if you don’t bother with the explanation clone away. Let’s start with the static page <code class="language-plaintext highlighter-rouge">index.html</code>, it’s basic html so there’s not to much to say</p>
<figure class="highlight"><pre><code class="language-html" data-lang="html"><span class="cp"><!DOCTYPE html></span>
<span class="nt"><html></span>
<span class="nt"><head></span>
<span class="nt"><meta</span> <span class="na">charset=</span><span class="s">"UTF-8"</span><span class="nt">></span>
<span class="nt"></head></span>
<span class="nt"><body></span>
<span class="nt"><h2></span>Hello World!<span class="nt"></h2></span>
<span class="nt"><h3></span>Containers for life!<span class="nt"></h3></span>
<span class="nt"></body></span>
<span class="nt"></html></span></code></pre></figure>
<p>To serve this page to someone, we need a web server to handle the request. I usually use Nginx since it does it all (reverse proxy, load balancer, mail proxy, e.t.c) and it’s super light weight. So we’ll use the convenient docker team supported image <code class="language-plaintext highlighter-rouge">nginx:alpine</code> which you can learn more about on <a href="https://hub.docker.com/_/nginx/">nginx’s dockerhub page</a>. I’ve also included an example nginx config file <code class="language-plaintext highlighter-rouge">default.conf</code> to overwrite the default’s within the base image.</p>
<figure class="highlight"><pre><code class="language-conf" data-lang="conf"><span class="n">server</span> {
<span class="n">listen</span> <span class="m">80</span>;
<span class="n">listen</span> [::]:<span class="m">80</span>;
<span class="n">server_name</span> <span class="n">localhost</span>;
<span class="n">location</span> / {
<span class="n">root</span> /<span class="n">usr</span>/<span class="n">share</span>/<span class="n">nginx</span>/<span class="n">html</span>;
<span class="n">index</span> <span class="n">index</span>.<span class="n">html</span> <span class="n">index</span>.<span class="n">htm</span>;
}
}</code></pre></figure>
<p>Here we can see that the server configuration is to listen on port 80 with the server name as localhost. If you’re deploying this to some server, you’d just had to change to the domain name that points there e.g. example.com. Since the only location is <code class="language-plaintext highlighter-rouge">/</code>, when you ask for <code class="language-plaintext highlighter-rouge">server_name:port</code>, it will treat the root folder on server as <code class="language-plaintext highlighter-rouge">/usr/share/nginx/html</code> and there it’ll serve you the the index file named <code class="language-plaintext highlighter-rouge">index.html</code> or <code class="language-plaintext highlighter-rouge">index.htm</code>. You can add other locations to point to other files in on your server (in this case it’s your container) by just adding new location blocks. Now that’s it for the server content and the nginx configuration. All we have to do is put it all together into a <code class="language-plaintext highlighter-rouge">Dockerfile</code>. So we’ll start with the base image <code class="language-plaintext highlighter-rouge">nginx:alpine</code>, and then copy the html file and config file into the container.</p>
<figure class="highlight"><pre><code class="language-dockerfile" data-lang="dockerfile"><span class="k">FROM</span><span class="s"> nginx:alpine</span>
<span class="k">COPY</span><span class="s"> default.conf /etc/nginx/conf.d/default.conf</span>
<span class="k">COPY</span><span class="s"> index.html /usr/share/nginx/html/index.html</span></code></pre></figure>
<p>That’s it for the container, we could build the image with <code class="language-plaintext highlighter-rouge">docker build -t static-web-page .</code>, but I usually prefer to use compose so I can simply call <code class="language-plaintext highlighter-rouge">docker-compose up -d</code> and call it a day. So we’ll add <code class="language-plaintext highlighter-rouge">docker-compose.yml</code></p>
<figure class="highlight"><pre><code class="language-docker" data-lang="docker">version: '2'
services:
static-web:
build: .
ports:
- "8000:80"</code></pre></figure>
<p>The basic highligths here are we’re using docker-compose version 2. The one service in the compse is called <code class="language-plaintext highlighter-rouge">static-web</code> and it is built using the previous <code class="language-plaintext highlighter-rouge">Dockerfile</code> with port 80 inside the container taking requests from 8000 outside. With this compose file, we can easily launch the server with <code class="language-plaintext highlighter-rouge">docker-compose up -d</code> and visit <code class="language-plaintext highlighter-rouge">localhost:8000</code> to get the html page we created at the beginning. Now any deploying this web page is one command from any computer with docker and docker-compose on it. Feel free to comment below if you have any questions.</p>Christopher J. AdkinsThese days I’ve been dockerizing everything, and love the fact that docker allows for a very transparent deployment process with the server management side of things. Currently I’m working for a Canadian telecommunications company, and updating some of their web development processes and pipelines using the power of docker. There is something satisfying about typing docker-compose up -d and that’s it (well, I guess I’ve set some of the older guys with a Portainer GUI, semantics!)Summation Machines2018-10-12T00:00:00+00:002018-10-12T00:00:00+00:00http://www.cjadkins.com/math/2018/10/12/summation-machines<p>A while ago I was watching some of <a href="http://physics.wustl.edu/cmb/">Carl Bender’s</a> physics lectures online and he was talking about the power of summation, specially summation of infinite sequence. There are many different ways to sum infinite sequences, some that make sense in typical way, and some that don’t. In any intro undergraduate class you’ll be introduced to the concept of infinite series, and you’ll learn about convergent and divergent series. Let’s say we have an infinite sequence \(\{a_n\}_{n\geq 0}\) where \(a_n\) lives in some ring \(R\), and we have the series</p>
\[\sum_{n=0}^\infty a_n = a_0 + a_1 + a_2 + a_3 + \ldots = \quad ?\]
<p>It always seemed strange to have the disconnect between the \(\Sigma\) and the left, and the \(+\)’s on the right. In a more general setting, you can play with this definition in a funny way with a few definition’s of what \(\Sigma\) can do. Let’s define an operator \(\Sigma\) that acts on an infinite sequence with the following properties.</p>
\[\sum \{a_n\}_{n\geq 0} = a_0 + \sum \{a_n\}_{n\geq 1} \label{1} \tag{seperation}\]
\[\sum \{b \cdot a_n\}_{n\geq 0} = b \cdot \sum \{a_n\}_{n\geq 0} \label{2} \tag{distributivity}\]
<p>These are fairly normal things to expect from your addition and multiplication operators. Quite easily now we can construct values of what the sum of some sequences are if we fix down some numbers. The first example most people are shown is the geometric series. We take \(a_n = a r^n\) where \(a,r \in \mathbb{C}\). We can easily see using the two properties above that</p>
\[\sum \{a r^n\}_{n\geq 0} = a + r \cdot \sum \{a r^n\}_{n\geq 0}\]
<p>Rearranging the equation and factoring the sum gives us the following equation</p>
\[\sum \{a r^n\}_{n\geq 0} = \frac {a}{1-r}\]
<p>Algebraically we’ve found a formula to give us a value for this infinite series. Setting aside the notation of convergence, we’ve now obtained a representation of the value of the sum. We can even do this to some sequences to assign a value to them under the operator \(\Sigma\). Take \(\{ (-1)^n \}_{n\geq 0}\). Via the formula we’ve derived, we now see (\(a=1,r=-1\))</p>
\[\sum \{ (-1)^n \}_{n\geq 0} = \frac{1}{2}\]
<p>Which is interesting since assuming associativity holds, we can group the sum in the following ways</p>
\[\sum \{ (-1)^n \}_{n\geq 0} = \underbrace{1 - 1}_{ =0} \underbrace{ +1 - 1}_{ =0} \underbrace{+1 - 1}_{ =0} + \ldots = 0 + 0 + 0 + \ldots =0\]
\[\sum \{ (-1)^n \}_{n\geq 0} = 1 +\underbrace{(-1 +1)}_{ =0} +\underbrace{(-1 +1)}_{ =0} + \ldots = 1 + 0 + 0 + \ldots = 1\]
<p>It would seem our summation operator \(\Sigma\) some how knew of the 2 possible values for the sum and decided to average them. You can even do this for sequences that don’t seem to converge to anything and get a representation of their sum. Take \(\{ 2^n \}_{n\geq 0}\), we see that (via the formula, \(a=1,r=2\))</p>
\[\sum \{ 2^n \}_{n\geq 0} = -1\]
<p>It seems that this divergent series has a happy value of \(-1\) that it wants to represent itself as. I’ll follow up this abstracted series stuff in another post later!</p>Christopher J. AdkinsA while ago I was watching some of Carl Bender’s physics lectures online and he was talking about the power of summation, specially summation of infinite sequence. There are many different ways to sum infinite sequences, some that make sense in typical way, and some that don’t. In any intro undergraduate class you’ll be introduced to the concept of infinite series, and you’ll learn about convergent and divergent series. Let’s say we have an infinite sequence \(\{a_n\}_{n\geq 0}\) where \(a_n\) lives in some ring \(R\), and we have the seriesThe Riemann Hypothesis solved?2018-10-04T00:00:00+00:002018-10-04T00:00:00+00:00http://www.cjadkins.com/math/2018/10/04/riemann-hypothesis-sovled<p>As you may or may not have heard, <a href="https://en.wikipedia.org/wiki/Michael_Atiyah">Michael Atiyah</a> has recently put out a short paper outlining his proof of the Riemann Hypothesis. As I was curious if it was true, I decided to check out <a href="/assets/posts/2018-The_Riemann_Hypothesis.pdf">his proof</a>. The initial doubt of the validity of this proof came about since you’ll see almost no reference to the properties of Zeta function itself. I won’t go into the gossip, of if it’s been solved or not, you can find that all over the internet.</p>
<p>Let’s have a quick crash course on the Riemann Zeta function. Back in the day Euler was studying the following power series:</p>
\[\zeta(s) = 1 + \frac{1}{2^s} + \frac{1}{3^s} + \frac{1}{4^s} + \frac{1}{5^s} + \ldots = \sum_{n=1}^\infty \frac{1}{n^s}\]
<p>where \(s>1\) since \(s=1\) is every calculus students favourite series, the harmonic series, which diverges. He also noticed that this series related to the prime number in a peculiar way, namely he found the following form of this series.</p>
\[\zeta(s) = \prod_{p} \frac{1}{1-p^{-s}}\]
<p>Which you can deduce looking at \(\zeta(s)\) and \(2^{-s} \zeta(s)\), and subtracting them to see:</p>
\[\left ( 1 - 2^{-s} \right) \zeta(s) = \underbrace{1 + \frac{1}{3^s} + \frac{1}{5^s} + \frac{1}{7^s} + \frac{1}{9^s} + \ldots}_{\text{no factors of 2 in denominator}}\]
<p>Continuing the pattern, we can remove the factors of 3 with the same trick:</p>
\[\left ( 1 - 3^{-s} \right) \left ( 1 - 2^{-s} \right) \zeta(s) = \underbrace{1 + \frac{1}{5^s} + \frac{1}{7^s} + \frac{1}{11^s} + \frac{1}{13^s} + \ldots}_{\text{no factors of 2 or 3 in denominator}}\]
<p>If we continued this using prime numbers, we can now easily see (maybe using the fundamental theorem of arithmetic for reference)</p>
\[\zeta(s) \prod_p \left ( 1 - p^{-s}\right ) = 1\]
<p>History moved on and complex analysis became gave mathematicians the ability to turn cheat codes on. People were able to analytically extend functions to the complex plane \(\mathbb{C}\) and Riemann found a helpful functional equation to extend \(\zeta\) of the real line:</p>
\[\zeta(s) = 2^s \pi^{s-1} \sin \left ( \frac{ \pi s }{2} \right ) \Gamma(1-s) \zeta(1-s)\]
<p>where \(\Gamma\) is the Gamma function (generalization of the factorial, \(\Gamma(n) = (n-1)!\) when \(n \in \mathbb{N}\)). If you’ve ever studied complex analysis, you’ll know that the zeros of a analytic function are the bread of butter to understanding it. Aside from the trivial zeros of sine in the functional equation, the goal was to find out the zeros. Riemann hypothesised that all the zeros lie on the line of \(s= \frac{1}{2} + i y\). Why that line? Well due to the symmetric under tones of the functional equation!</p>Christopher J. AdkinsAs you may or may not have heard, Michael Atiyah has recently put out a short paper outlining his proof of the Riemann Hypothesis. As I was curious if it was true, I decided to check out his proof. The initial doubt of the validity of this proof came about since you’ll see almost no reference to the properties of Zeta function itself. I won’t go into the gossip, of if it’s been solved or not, you can find that all over the internet.A little bit of jekyll in the morning2018-09-07T00:00:00+00:002018-09-07T00:00:00+00:00http://www.cjadkins.com/random/2018/09/07/a-little-bit-of-jekyll<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">print_hi</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="n">puts</span> <span class="s">"Hi, #{name}"</span>
<span class="n">end</span>
<span class="n">print_hi</span><span class="p">(</span><span class="s">'Tom'</span><span class="p">)</span>
<span class="c1">#=> prints 'Hi, Tom' to STDOUT.</span></code></pre></figure>
<h3 id="tables">Tables</h3>
<table>
<thead>
<tr>
<th>Title 1</th>
<th>Title 2</th>
<th>Title 3</th>
<th>Title 4</th>
</tr>
</thead>
<tbody>
<tr>
<td>lorem</td>
<td>lorem ipsum</td>
<td>lorem ipsum dolor</td>
<td>lorem ipsum dolor sit</td>
</tr>
<tr>
<td>lorem ipsum dolor sit</td>
<td>lorem ipsum dolor sit</td>
<td>lorem ipsum dolor sit</td>
<td>lorem ipsum dolor sit</td>
</tr>
<tr>
<td>lorem ipsum dolor sit</td>
<td>lorem ipsum dolor sit</td>
<td>lorem ipsum dolor sit</td>
<td>lorem ipsum dolor sit</td>
</tr>
<tr>
<td>lorem ipsum dolor sit</td>
<td>lorem ipsum dolor sit</td>
<td>lorem ipsum dolor sit</td>
<td>lorem ipsum dolor sit</td>
</tr>
</tbody>
</table>
<table>
<thead>
<tr>
<th>Title 1</th>
<th>Title 2</th>
<th>Title 3</th>
<th>Title 4</th>
</tr>
</thead>
<tbody>
<tr>
<td>lorem</td>
<td>lorem ipsum</td>
<td>lorem ipsum dolor</td>
<td>lorem ipsum dolor sit</td>
</tr>
<tr>
<td>lorem ipsum dolor sit amet</td>
<td>lorem ipsum dolor sit amet consectetur</td>
<td>lorem ipsum dolor sit amet</td>
<td>lorem ipsum dolor sit</td>
</tr>
<tr>
<td>lorem ipsum dolor</td>
<td>lorem ipsum</td>
<td>lorem</td>
<td>lorem ipsum</td>
</tr>
<tr>
<td>lorem ipsum dolor</td>
<td>lorem ipsum dolor sit</td>
<td>lorem ipsum dolor sit amet</td>
<td>lorem ipsum dolor sit amet consectetur</td>
</tr>
</tbody>
</table>
<blockquote>
<p>This quote will change your life. It will reveal the secrets of the universe, and all the wonders of humanity. Don’t misuse it.</p>
</blockquote>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><html></span>
<span class="nt"><head></span>
<span class="nt"></head></span>
<span class="nt"><body></span>
<span class="nt"><p></span>Hello, World!<span class="nt"></p></span>
<span class="nt"></body></span>
<span class="nt"></html></span>
</code></pre></div></div>Christopher J. Adkinsdef print_hi(name) puts "Hi, #{name}" end print_hi('Tom') #=> prints 'Hi, Tom' to STDOUT.What is a Spin Glass?2015-05-12T00:00:00+00:002015-05-12T00:00:00+00:00http://www.cjadkins.com/math/2015/05/12/what-is-a-spin-glass<p>In an attempt to motivate myself into keeping this updated, I’ll try to write a few notes on what spin glasses are and why we care. Our first example will be glass (mostly SiO2), the following is a snapshot of its atomic structure.</p>
<p class="center"><img src="/assets/posts/glass.png" alt="glass" /></p>
<p>As you can see the structure is irregular and looks slightly random. Why is this the case? It’s because not all atoms are happy with their magnetic spin alignment! The atoms are forced into a frustrated position from the other atoms in the system (I’m vague on “other “atoms because this is one of our key choices in the working model). It may be better to call these objects frustrated magnets, since that’s what they are, though the name spin glass comes from trying to understand the structure of atoms with spins frozen in place (like glass!)</p>
<p>To begin our study, we need a basis for the model of the interaction of these atoms. The most famous to date(that I know of) is the Sherrington-Kirckpatrick Model. The model assumes that the energy (Hamiltonian,\(H\)) is given by the sum 2-way interactions between particles. Assume we have \(N\) particles in the system, each particle has spin up ( \(\sigma_i=+1\)) or spin down ( \(\sigma_i=-1\) ) so \(\sigma \in \{-1,1\}^N\) , and the interaction (Coupling Force) between them is given by \(J_{i,j}\). In addition, we don’t want to work with quantities that are too large, so we’ll normalize the square to constant order. This gives</p>
\[H_N( \sigma) := \frac{1}{\sqrt{N}} \sum_{i , j } J_{i,j} \sigma_i \sigma_j\]
<p>The interaction is typically chosen as a standard Gaussian random variable. i.e. something with \(\mathbb{E} g = 0\) (mean 0) and \(\mathbb{E} g^2 = 1\) (variance 1). It turns out that the distribution of the interactions always tends to standard Gaussian as we increase the system size \(N\) (think central limit theorem modulo the technical details), so this assumption is for simplicity of computations. With this we may easily compute the covariance of the Hamiltonian (i.e. similarity between two different spin states \(\sigma^1\) and \(\sigma^2\) )</p>
\[R_{1,2} : =\mathbb{E} H_N ( \sigma^1 ) H_N ( \sigma^2) =\frac{1}{N} \sum_{i=1}^N \sigma_i^1\sigma_i^2 = \frac{ \sigma^1 \cdot \sigma^2 }{N}\]
<p>This quantity will repeated arise and as of such it is a key quantity. It is called the overlap between \(\sigma^1\) and \(\sigma^2\). We see that it is a measure of how similar the states, with \(R_{1,2}=1\) for states being the same and \(R_{1,2}=-1\) for being completely different. It turns out that \(R_{1,2} \geq 0\) with high probability, otherwise known as Talagrand’s Positivity Principal. This isn’t really of importance at the present moment, but it’s an interesting fact to mention here. To brush by some of the physics behind the statistical mechanics at work here, we assign probability weights to a spin state \(\sigma\) via the Boltzmann distribution, i.e.</p>
\[\mathbb{P}( \sigma\in \Sigma^N) \propto \exp ( \beta H_N(\sigma))\]
<p>where \(\beta\) is the inverse temperature of the state(up to a factor known as the Boltzmann constant). We can create a measure on the states, call a Gibb’s measure, by normalizing over the weight of all states. We’ll call this normalization a Partition Function:</p>
\[Z_N = \sum_{\sigma \in \Sigma^N} \exp ( \beta H_N ( \sigma) )\]
<p>We now have a perfectly good probability measure to work with now, the Gibb’s measure.</p>
\[\mathbb{P}( \sigma\in \Sigma^N) = \frac{\exp ( \beta H_N(\sigma))}{Z_N}\]
<p>In addition to this, it’s helpful to define a Gibb’s average (an average with respect to the Gibb’s measure) for function on \(\Sigma^N\) by</p>
\[\langle f(\sigma) \rangle = \sum_{ \sigma \in \Sigma^N} \frac{f( \sigma ) \exp ( \beta H_N( \sigma))}{Z_N}\]
<p>The big question that has gotten a great deal of attention is computing the Free Energy of the limiting system ( when \(N \to \infty\), i.e. the number of atoms becomes very large). The Free Energy is defined as</p>
\[F_N = \frac{1}{N} \mathbb{E} \log Z_N\]
<p>It turns out that this quantity can tell us about the best energy state of the system (which is why it is the big question!), so let’s see how. We’d like to understand</p>
\[\lim_{N \to \infty} \frac{1}{N} \mathbb{E} \max_{ \sigma \in \Sigma^n} H_N( \sigma)\]
<p>One may notice that we may easily bound the free energy over the inverse temperature with this quantity by keeping only the largest \(H_N\) and replacing every state with the largest \(H_N\), i.e.</p>
\[\lim_{N \to \infty} \frac{1}{N} \mathbb{E} \max_{ \sigma \in \Sigma^n} H_N( \sigma) \leq \lim_{N \to \infty} \frac{1}{N \beta} \mathbb{E} \log Z_N(\sigma) \leq \frac{ \log2 }{ \beta }\ + \lim_{N \to \infty} \frac{1}{N} \mathbb{E} \max_{ \sigma \in \Sigma^n} H_N( \sigma)\]
<p>Thus the difference between the two is just \(\log 2 / \beta\), so if we denote \(F( \beta) = \lim_{N \to \infty} F_N( \beta)\) we see</p>
\[\lim_{\beta \to \infty} \frac{ F(\beta)}{\beta} = \lim_{N \to \infty} \frac{1}{N} \mathbb{E} \max_{ \sigma \in \Sigma^n} H_N( \sigma)\]
<p>It turns out that \(F(\beta)\) can be calculated and is known as the Parisi Formula.</p>Christopher J. AdkinsIn an attempt to motivate myself into keeping this updated, I’ll try to write a few notes on what spin glasses are and why we care. Our first example will be glass (mostly SiO2), the following is a snapshot of its atomic structure.