-
Notifications
You must be signed in to change notification settings - Fork 0
/
flask-in-a-rump-kernel.html
235 lines (221 loc) · 13.7 KB
/
flask-in-a-rump-kernel.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
<!DOCTYPE html>
<html lang="en">
<head>
<link href='//fonts.googleapis.com/css?family=Source+Sans+Pro:300,400,700,400italic' rel='stylesheet' type='text/css'>
<link rel="stylesheet" type="text/css" href="http://projects.curiousllc.com/theme/css/style.min.css">
<link rel="stylesheet" type="text/css" href="http://projects.curiousllc.com/theme/css/pygments.min.css">
<link rel="stylesheet" type="text/css" href="http://projects.curiousllc.com/theme/css/font-awesome.min.css">
<link href="http://projects.curiousllc.com/feeds/all.atom.xml" type="application/atom+xml" rel="alternate" title="Working with hardware Atom">
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="robots" content="" />
<meta name="author" content="ryanday" />
<meta name="description" content="Running Python on a Rump Kernel" />
<meta name="keywords" content="rumpkernel, python, flask">
<meta property="og:site_name" content="Working with hardware"/>
<meta property="og:title" content="Flask in a Rump Kernel"/>
<meta property="og:description" content="Running Python on a Rump Kernel"/>
<meta property="og:locale" content="en_US"/>
<meta property="og:url" content="http://projects.curiousllc.com/flask-in-a-rump-kernel.html"/>
<meta property="og:type" content="article"/>
<meta property="article:published_time" content="2015-07-28 11:15:00-04:00"/>
<meta property="article:modified_time" content=""/>
<meta property="article:author" content="http://projects.curiousllc.com/author/ryanday.html">
<meta property="article:section" content="os"/>
<meta property="article:tag" content="rumpkernel"/>
<meta property="article:tag" content="python"/>
<meta property="article:tag" content="flask"/>
<meta property="og:image" content="https://avatars3.githubusercontent.com/u/6845642?v=3&s=200"> <title>Working with hardware – Flask in a Rump Kernel</title>
</head>
<body>
<aside>
<div>
<a href="http://projects.curiousllc.com">
<img src="https://avatars3.githubusercontent.com/u/6845642?v=3&s=200" alt="" title="">
</a>
<h1><a href="http://projects.curiousllc.com"></a></h1>
<p></p>
<nav>
<div style="text-align: left; font-size: 0.9em">At Curious, we connect Things to the Cloud. <br/><br/>
If you would like to launch a product, or connect your product to the Cloud, drop us a line!</div>
<ul class="list">
<li><a href="mailto:[email protected]" target="_blank">[email protected]</a></li>
<li><a href="http://curiousllc.com/" target="_blank">Curious LLC</a></li>
<li><a href="http://www.meetup.com/NOVA-Python" target="_blank">Nova Python</a></li>
<li><a href="http://www.meetup.com/Golang-DC/" target="_blank">Golang DC</a></li>
</ul>
</nav>
<ul class="social">
<li><a class="sc-You can add links in your config file" href="#" target="_blank"><i class="fa fa-You can add links in your config file"></i></a></li>
<li><a class="sc-Another social link" href="#" target="_blank"><i class="fa fa-Another social link"></i></a></li>
</ul>
</div>
</aside>
<main>
<article>
<header>
<h1 id="flask-in-a-rump-kernel">Flask in a Rump Kernel</h1>
<p>
Posted on Tue 28 July 2015 in <a href="http://projects.curiousllc.com/category/os.html">os</a> by <a href="http://projects.curiousllc.com/author/ryanday.html">ryanday</a>
• Tagged with
<a href="http://projects.curiousllc.com/tag/rumpkernel.html">rumpkernel</a>, <a href="http://projects.curiousllc.com/tag/python.html">python</a>, <a href="http://projects.curiousllc.com/tag/flask.html">flask</a>
</p>
</header>
<div>
<div class="section" id="rump-kernels">
<h2>Rump Kernels</h2>
<p><em>[Updated for Python 3.5.1]</em></p>
<p><a class="reference external" href="https://github.com/rumpkernel/wiki/wiki/Info:-FAQ#user-content-What_is_a_rump_kernel">Rump Kernels</a> are better explained by the FAQ than myself, so I suggest reading
through it. But essentially a <strong>Rump Kernel</strong> is a kernel with everything but your bare
essentials removed. Don't need a file system? Then don't bundle the file system
drivers. What about the drivers you do need? You get them from an <strong>Anykernel</strong>!</p>
<p>The idea of an <strong>Anykernel</strong> is that individual drivers can be taken and used in a Rump
Kernel. NetBSD is an Anykernel. What this means is that a Rump Kernel can use the
drivers directly from the NetBSD source tree. You don't need to build your own drivers
to use a Rump Kernel. You can use real-world drivers to build production ready
<strong>Unikernels</strong> (<em>wait... unikernels?</em>).</p>
<p>At this time, a <strong>Unikernel</strong> is built with a script called Rumprun. Rumprun takes your
application, libc, the rumpkernel, and hypervisor layer, then builds the final product
(unikernel) that will interact with things. The Unikernel is what you end up running in
Xen or QEMU.</p>
<p><em>So... back to Python remember?</em></p>
<p>Yes. A Unikernel provides you with a "POSIX-y" environment to operate in. This lets
most software run right out of the box. Which means we can probably get Python up
and running without too much trouble.</p>
<p>In fact, Python already exists in the Rumprun <a class="reference external" href="https://github.com/rumpkernel/rumprun-packages/tree/master/python">package system</a>.</p>
</div>
<div class="section" id="the-python-package">
<h2>The Python Package</h2>
<p>The Rumprun <a class="reference external" href="https://github.com/rumpkernel/rumprun-packages">package system</a> is similar to the BSD ports structure. A source
distribution is specified and downloaded. Patches are applied to the source. Then
everything is compiled into a Unikernel.</p>
<p>In the case of Python there were a couple hurdles to overcome.</p>
<ul class="simple">
<li><strong>Cross Compilation</strong>. The build-rr toolchain lets us build binaries for the Rump
Kernel environment. The Python config scripts need to be patched in order to make
this cross compilation option available.</li>
<li><strong>Static builds</strong>. Python needs to be compiled statically. We are creating a single,
bootable, Unikernel. We can't rely on dynamic libraries being availble at runtime.</li>
<li><strong>Bootstrapping Python</strong>. Python actually needs Python to build Python. This is
problematic when we are cross compiling. The target Python (and tools) that we
are building can't run on the host platform. This is solved by building Python
twice. The first time we build for the host, and save the required binaries.
The second time we cross build for the target, and use the host binaries to help.</li>
</ul>
<p>The Python package makes these cross compilation updates. It also modifies the
<strong>Modules/Setup.dist</strong> file to build several modules statically along with Python.</p>
<p>To build the Python package you will need a working toolchain, <a class="reference external" href="https://github.com/rumpkernel/wiki/wiki/Tutorial%3A-Building-Rumprun-Unikernels">start here</a>. I will assume that you have a working build environment for the rest of the tutorial.</p>
<p>Now grab the rumprun packages repo, enter the python package, and start the build.</p>
<div class="highlight"><pre>git clone [email protected]:rumpkernel/rumprun-packages.git
<span class="nb">cd </span>python
<span class="nb">export </span><span class="nv">PATH</span><span class="o">=</span>/path/to/app-tools:<span class="nv">$PATH</span>
make
</pre></div>
<p>You're going to see a lot of errors like this one:</p>
<div class="highlight"><pre>collect2: error: ld returned 1 <span class="nb">exit </span>status
</pre></div>
<p>But you will also end up with a Python ISO in the images/ directory. To test out
your Python package installation, please follow the directions in the <a class="reference external" href="https://github.com/rumpkernel/rumprun-packages/tree/master/python">README.md</a>
file. This will walk you through rumpbake and how to build the bootstrap Python script.</p>
</div>
<div class="section" id="running-flask">
<h2>Running Flask</h2>
<p>At this point, you should have a Unikernel booting up and running the Python "Hello World"
example. Now our goal is to:</p>
<ul class="simple">
<li>Build a Python environment with the modules we need</li>
<li>Include that environment in our unikernel</li>
<li>Setup networking</li>
</ul>
<p>Building the Python environment is fairly straight forward. Using the same version of Python
as your Unikernel (probably 3.4.3) create a virtual environment and install any packages
you'd like available to your Unikernel. In this case we're just installing Flask.</p>
<div class="highlight"><pre>pyvenv-3.5 helloworld-env
<span class="nb">source </span>helloworld-env/bin/activate
pip install flask
</pre></div>
<p>To include this environment in the Unikernel, we want to build an ISO containing our site
packages. We will end up mounting this ISO inside the Unikernel. Just like we did for
the static Python build.</p>
<div class="highlight"><pre>genisoimage -r -o py3env.iso helloworld-env/lib/python3.5/site-packages
</pre></div>
<p>Setting up networking requires us to create a TAP device. This will let the host machine
pass traffic to our Unikernel. We'll setup 10.0.120.100, and then our Unikernel will
listen on 10.0.120.101.</p>
<div class="highlight"><pre>sudo ip tuntap add tap0 mode tap
sudo ip addr add 10.0.120.100/24 dev tap0
sudo ip link <span class="nb">set </span>dev tap0 up
</pre></div>
<p>We will avoid doing anything special for our Flask application. Just a basic example will
be enough for now.</p>
<div class="highlight"><pre><span class="c"># main.py</span>
<span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">Flask</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span><span class="n">__name__</span><span class="p">)</span>
<span class="nd">@app.route</span><span class="p">(</span><span class="s">"/"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">hello</span><span class="p">():</span>
<span class="k">return</span> <span class="s">"Hello World!"</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">"__main__"</span><span class="p">:</span>
<span class="n">app</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">host</span><span class="o">=</span><span class="s">'0.0.0.0'</span><span class="p">)</span>
</pre></div>
<p>Now we are able to start our Unikernel. The following script should be familiar. We are
following the steps from the README.md file from earlier. However we also setup networking
and mount our Python environment in the site-packages directory that our Unikernel sees.</p>
<div class="highlight"><pre><span class="c">#!/bin/sh</span>
<span class="nv">PATH</span><span class="o">=</span>/home/rday/Development/os/rump/qemu/bin:<span class="nv">$PATH</span>:/home/rday/Development/os/rump/rumprun/app-tools
cython --embed -v -3 -Werror main.py
x86_64-rumprun-netbsd-gcc main.c -o main.o <span class="se">\</span>
-I../Python-3.5.1/pythondist/include/python3.5m <span class="se">\</span>
-L../Python-3.5.1/pythondist/lib <span class="se">\</span>
-lpython3.5m -lutil -lm -lz -lssl -lcrypto
rumprun-bake hw_virtio main.bin main.o
rumprun qemu -i <span class="se">\</span>
-I <span class="k">if</span>,vioif,<span class="s1">'-net tap,ifname=tap0,script=no'</span><span class="se">\</span>
-W <span class="k">if</span>,inet,static,10.0.120.101/24 <span class="se">\</span>
-b ../../python.iso,/python/lib/python3.5 <span class="se">\</span>
-b ../../stubetc.iso,/etc <span class="se">\</span>
-b py3env.iso,/python/lib/python3.5/site-packages <span class="se">\</span>
-e <span class="nv">PYTHONHOME</span><span class="o">=</span>/python main.bin
</pre></div>
<p>Give it a shot. Run the script, and visit <a class="reference external" href="http://10.0.120.101:5000">http://10.0.120.101:5000</a>. You should get the
"Hello World" message to pop up!</p>
</div>
</div>
<div class="tag-cloud">
<p>
<a href="http://projects.curiousllc.com/tag/rumpkernel.html">rumpkernel</a>
<a href="http://projects.curiousllc.com/tag/python.html">python</a>
<a href="http://projects.curiousllc.com/tag/flask.html">flask</a>
</p>
</div>
</article>
<footer>
<p>© Ryan Day </p>
<p>Built using <a href="http://getpelican.com" target="_blank">Pelican</a> - <a href="https://github.com/alexandrevicenzi/flex" target="_blank">Flex</a> theme by <a href="http://alexandrevicenzi.com" target="_blank">Alexandre Vicenzi</a></p> </footer>
</main>
<script type="text/javascript">
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-2393416-13', 'auto');
ga('send', 'pageview');
</script>
<script type="application/ld+json">
{
"@context": "http://schema.org",
"@type": "BlogPosting",
"name": "Flask in a Rump Kernel",
"headline": "Flask in a Rump Kernel",
"datePublished": "2015-07-28 11:15:00-04:00",
"dateModified": "",
"author": {
"@type": "Person",
"name": "ryanday",
"url": "http://projects.curiousllc.com/author/ryanday.html"
},
"image": "https://avatars3.githubusercontent.com/u/6845642?v=3&s=200",
"url": "http://projects.curiousllc.com/flask-in-a-rump-kernel.html",
"description": "Running Python on a Rump Kernel"
}
</script></body>
</html>