tag:blogger.com,1999:blog-25284548345649011502024-02-08T02:36:44.600-08:00Cloudy Stuff HappensUnknownnoreply@blogger.comBlogger3125tag:blogger.com,1999:blog-2528454834564901150.post-82142705743866364122013-04-10T20:50:00.000-07:002013-04-10T20:50:17.636-07:00Understanding nova-conductor in OpenStack Nova<br />
<h1 id="markdown-header-understanding-nova-conductor-in-openstack-nova" style="background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 24px; font-weight: normal; line-height: 1.25; margin: 0px 0px 10px; padding: 0px;">
</h1>
<h1 id="markdown-header-understanding-nova-conductor-in-openstack-nova" style="font-size: 24px; font-weight: normal; line-height: 1.25; margin: 0px 0px 10px; padding: 0px;">
Understanding nova-conductor in OpenStack Nova</h1>
<div style="font-size: 14px; line-height: 20px; margin-top: 10px;">
Nova conductor[<a href="http://russellbryantnet.wordpress.com/2012/11/19/a-new-nova-service-nova-conductor/" style="color: #3b73af; text-decoration: none;">1</a>,<a href="http://www.danplanet.com/blog/2013/02/07/all-your-db-are-belong-to-conductor/" style="color: #3b73af; text-decoration: none;">2</a>] is a new service in Nova introduced in the Grizzly release. Personally I would put it as one of the top three changes in Nova in the Grizzly cycle, along with bare metal provisioning and cell. In this article, I'd like to discuss its pros and cons to help you understand it better.</div>
<h2 id="markdown-header-what-is-it" style="font-size: 20px; font-weight: normal; line-height: 1.5; margin: 20px 0px 0px;">
What is it?</h2>
<div style="font-size: 14px; line-height: 20px; margin-top: 10px;">
nova-conductor is a RPC server. Check out <code style="background-color: whitesmoke; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(204, 204, 204); font-size: 12px; line-height: 1.4; padding: 1px 3px;">nova.conductor.rpcapi.ConductorAPI</code> to see the precise list of the APIs it supports. It is stateless, and is horizontally scalable, meaning that you can start as many instances on many servers as you want. The majority of the APIs are database proxy calls. Some are proxy calls to other RPC servers such as nova-api and nova-network. The client side of the RPC call is inside nova-compute. For example, if there is a need to update certain state of a VM instance in nova-compute, instead of connecting to the database directly, make a RPC call to nova-conductor, which connects to the database and makes the actual db update.</div>
<h2 id="markdown-header-should-i-use-it" style="font-size: 20px; font-weight: normal; line-height: 1.5; margin: 20px 0px 0px;">
Should I use it?</h2>
<div style="font-size: 14px; line-height: 20px; margin-top: 10px;">
Let's examine some pros and cons of conductor and you be the judge.</div>
<h3 id="markdown-header-security" style="font-size: 16px; line-height: 1.5625; margin: 20px 0px 5px;">
Security</h3>
<ul style="font-size: 14px; line-height: 20px; margin: 10px 0px 0px;">
<li>Benefit: We know each compute node runs a nova-compute service daemon. Before Grizzly, every nova-compute service has direct access to the database. If, for whatever reason, one of your compute nodes is compromized, then the attacker has (almost) full access to the database. With nova-conductor, the compromised node is limited to the extend that the conductor APIs allow it to do.</li>
<li>Limitation: First, although the conductor APIs limit what you can do against the database, it is still fairly permissive. For example, you can list all instances in the db, update their state, and destroy them, not just to the instances running on the compute node. Second, even if nova-compute has no direct database access, other services running on the same compute node might still do. If you use nova network with the multi-host mode, then you have nova-compute, nova-api-metadata and nova-network daemons running on every compute node. Unfortunately both nova-api-metadata and nova-network services need to have direct database access to work, so securing nova-compute alone doesn't help as much.</li>
</ul>
<h3 id="markdown-header-ease-of-upgrade" style="font-size: 16px; line-height: 1.5625; margin: 20px 0px 5px;">
Ease of Upgrade</h3>
<ul style="font-size: 14px; line-height: 20px; margin: 10px 0px 0px;">
<li>Benefit: The conductor decouples nova-compute and the database. So, if the database schema is upgraded, it's not necessary to upgrade nova-compute at the same time, as long as nova-conductor remains API compatible. This is a very important step towards rolling upgrade.</li>
<li>Limitation: Currently rolling upgrade is not yet ready in Grizzly.</li>
</ul>
<h3 id="markdown-header-performance" style="font-size: 16px; line-height: 1.5625; margin: 20px 0px 5px;">
Performance</h3>
<ul style="font-size: 14px; line-height: 20px; margin: 10px 0px 0px;">
<li>Benifit: By default, a database call is blocking in Nova. This is because nova-compute only has one OS thread, and use green threads in eventlet to achieve parallelism. The mysql driver written as as a C module doesn't respect green threads. That means, if you have 10 green threads to work on 10 instances, but the first scheduled green thread is blocked on a db call, the other 9 threads must wait. With nova conductor, there is no db calls, but RPC calls. Guess what, RPC calls are green thread friendly so the other 9 threads can continue while the first thread waiting for the RPC call to complete. This would increase the parallelism. If you are running many instances per compute node, it could lower your average response time.</li>
<li>Limitation: Ironically, conductor could also increase response time. First, because instead of doing a direct db call, we need to call it via RPC, which is one more indirection, or two indirections if you use RabbitMQ/Qpid. That means longer latency. Second, the db calls are still blocking on nova-conductor. So, if you only have 1 nova-conductor daemon running, you essentially are serializing all db calls initiated from compute nodes. That'd most likely be a problem in a large deployment. Make sure you select the right number of nova-conductor accordindly.</li>
</ul>
<div style="font-size: 14px; line-height: 20px; margin-top: 10px;">
OK. After weighing the pros and cons, hopefully you have made up your own mind. Also keep in mind that the limitations only apply to the current implementation (Grizzly stable release). I'm sure there are work in the pipeline to address those. If you are still undecided, let me try to make the decision process simpler for you: <strong>If you are upgrading from Folsom and run nova-network in multi-host mode, don't use nova-conductor, otherwise do</strong>.</div>
<h2 id="markdown-header-how-to-not-use-it" style="font-size: 20px; font-weight: normal; line-height: 1.5; margin: 20px 0px 0px;">
How to (not) use it?</h2>
<div style="font-size: 14px; line-height: 20px; margin-top: 10px;">
If you decide to use nova-conductor, read this <a href="http://russellbryantnet.wordpress.com/2013/02/19/deployment-considerations-for-nova-conductor-service-in-openstack-grizzly/" style="color: #3b73af; text-decoration: none;">blog post</a> to learn its deployment.</div>
<div style="font-size: 14px; line-height: 20px; margin-top: 10px;">
If you decide not to, put the following in your <code style="background-color: whitesmoke; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(204, 204, 204); font-size: 12px; line-height: 1.4; padding: 1px 3px;">nova.conf</code> on each compute node:</div>
<div class="codehilite" style="font-size: 14px; line-height: 20px;">
<pre style="background-color: whitesmoke; border-bottom-left-radius: 5px; border-bottom-right-radius: 5px; border-top-left-radius: 5px; border-top-right-radius: 5px; border: 1px solid rgb(204, 204, 204); box-sizing: border-box; font-family: 'Bitstream Vera Sans Mono', 'DejaVu Sans Mono', Monaco, monospace; font-size: 12px; line-height: 1.4; margin-bottom: 9px; margin-top: 9px; overflow: auto; padding: 10px; width: 938px;"><span class="k" style="color: #004080;">[conductor]</span>
<span class="na" style="color: teal;">use_local</span><span class="o">=</span><span class="s" style="color: #bb8844;">true</span>
</pre>
</div>
<div style="font-size: 14px; line-height: 20px; margin-top: 10px;">
Enjoy!</div>
Unknownnoreply@blogger.com8tag:blogger.com,1999:blog-2528454834564901150.post-43384526387811236442013-04-08T10:28:00.000-07:002013-04-29T08:30:28.430-07:00How to run pylint with few false positives<h1 id="markdown-header-how-to-run-pylint-with-few-false-positives" style="background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 24px; font-weight: normal; line-height: 1.25; margin: 0px 0px 10px; padding: 0px;">
How to run pylint with few false positives</h1>
<h2 id="markdown-header-introduction" style="background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 20px; font-weight: normal; line-height: 1.5; margin: 10px 0px 0px;">
Introduction</h2>
<div style="background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
Python is a fantastic programming language. It's succinct, easily readable and great for a lot of purposes. For example, all <a href="http://www.openstack.org/" style="color: #3b73af; text-decoration: none;">OpenStack</a> projects so far are written in Python. However, it's an interpreted language. There is no (default) compiler to do any static analysis such as type checking for you. The code quality heavily depends on code reviews (i.e. human eyes and brains) and test coverage.</div>
<div style="background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
There are several projects, such as pylint, pychecker and pyflakes, that do static code analysis to find bugs and coding style violations. Among them, pylint is the most sophisticated one. Unfortunately, Python code doesn't have any type annotations. It's a dynamically typed language. Therefore, Python type analysis is a really hard problem. In pylint, you <em>will</em> see false positives, meaning it will complain some lines as bugs that are actually correct. In fact, pylint was used early in OpenStack Nova's testing environment (around 2010) but latter dropped because the growing number of false positives became unmanagable.</div>
<h2 id="markdown-header-meet-lintstack" style="background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 20px; font-weight: normal; line-height: 1.5; margin: 20px 0px 0px;">
Meet lintstack</h2>
<div style="background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
lintstack is designed to address this problem: <strong>reduce false positives from pylint as much as possible without sacrificing accuracy</strong>. Certainly it's possible to improve pylint itself and perhaps type inference theory behind it but I'd leave that to the programming language researchers.</div>
<h3 id="markdown-header-key-idea" style="background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 16px; line-height: 1.5625; margin: 20px 0px 5px;">
Key idea</h3>
<div style="background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
The key idea is to leverage the version control system. I use git in the article but it works on others too. In git, you have access to the entire commit history of the project, but pylint only runs against the latest commit, or HEAD. Can we do better?</div>
<div style="background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
Let me use an example. Suppose I clone the project in git, HEAD is a <code style="background-color: whitesmoke; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(204, 204, 204); font-size: 12px; line-height: 1.4; padding: 1px 3px;">A</code>. I write a new feature, commit the patch, now the HEAD is at <code style="background-color: whitesmoke; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(204, 204, 204); font-size: 12px; line-height: 1.4; padding: 1px 3px;">B</code>. I run pylint against <code style="background-color: whitesmoke; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(204, 204, 204); font-size: 12px; line-height: 1.4; padding: 1px 3px;">B</code>, and see 60 errors in the report. That's an overwelming number. Which errors are due to my patch? If there were no false positives and commit <code style="background-color: whitesmoke; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(204, 204, 204); font-size: 12px; line-height: 1.4; padding: 1px 3px;">A</code> has no errors, then all 60 errors must all belong to my patch. But in reality there are false positives and real bugs in the commit <code style="background-color: whitesmoke; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(204, 204, 204); font-size: 12px; line-height: 1.4; padding: 1px 3px;">A</code>, so it's hard to tell which ones I should fix.</div>
<div style="background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
At a high level lintstack runs pylint on both commits <code style="background-color: whitesmoke; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(204, 204, 204); font-size: 12px; line-height: 1.4; padding: 1px 3px;">A</code> and <code style="background-color: whitesmoke; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(204, 204, 204); font-size: 12px; line-height: 1.4; padding: 1px 3px;">B</code>, and do a diff. lintstack considers errors on <code style="background-color: whitesmoke; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(204, 204, 204); font-size: 12px; line-height: 1.4; padding: 1px 3px;">A</code> are all false positives. Only the new errors are attributed to the patch. It might still be likely that some of the new errors are false positives, but it's much more manageable. Because the patch is small, I will only likely to see one or two errors instead of 60. I will pay attentions to the errors and fix accordingly.</div>
<h3 id="markdown-header-details" style="background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 16px; line-height: 1.5625; margin: 20px 0px 5px;">
Details</h3>
<div style="background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
The idea of lintstack is really simple, but it requires a little more careful thoughts.</div>
<ul style="background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin: 10px 0px 0px;">
<li>First, how do you do diff? In the new patch, the code is changed. That means, the line numbers, white spaces, or line breaks might change. So the error report on commit <code style="background-color: whitesmoke; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(204, 204, 204); font-size: 12px; line-height: 1.4; padding: 1px 3px;">A</code> might need to be revised to match that on commit <code style="background-color: whitesmoke; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(204, 204, 204); font-size: 12px; line-height: 1.4; padding: 1px 3px;">B</code>.</li>
<li>Second, the false positives we learned from commit <code style="background-color: whitesmoke; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(204, 204, 204); font-size: 12px; line-height: 1.4; padding: 1px 3px;">A</code> might help to reduce that on commit <code style="background-color: whitesmoke; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(204, 204, 204); font-size: 12px; line-height: 1.4; padding: 1px 3px;">B</code>. For example, pylint always seems to think that the <code style="background-color: whitesmoke; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(204, 204, 204); font-size: 12px; line-height: 1.4; padding: 1px 3px;">sha1</code> module has no <code style="background-color: whitesmoke; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(204, 204, 204); font-size: 12px; line-height: 1.4; padding: 1px 3px;">digestsize</code> method. If we have seen that on commit <code style="background-color: whitesmoke; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(204, 204, 204); font-size: 12px; line-height: 1.4; padding: 1px 3px;">A</code>, and a new error from commit <code style="background-color: whitesmoke; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(204, 204, 204); font-size: 12px; line-height: 1.4; padding: 1px 3px;">B</code> is of the same kind, even thought it's new code, we should classify it as a false positive suppress them.</li>
<li>Last but not the least, the git history is a complicated graph. When I'm developing a patch, the master branch has moved forward because other developers make commits too! Say the master moves from <code style="background-color: whitesmoke; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(204, 204, 204); font-size: 12px; line-height: 1.4; padding: 1px 3px;">A</code> to <code style="background-color: whitesmoke; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(204, 204, 204); font-size: 12px; line-height: 1.4; padding: 1px 3px;">A'</code>, it's not enough to run diff between <code style="background-color: whitesmoke; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(204, 204, 204); font-size: 12px; line-height: 1.4; padding: 1px 3px;">A</code> and <code style="background-color: whitesmoke; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(204, 204, 204); font-size: 12px; line-height: 1.4; padding: 1px 3px;">B</code>. It should be <code style="background-color: whitesmoke; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(204, 204, 204); font-size: 12px; line-height: 1.4; padding: 1px 3px;">A'</code> and<code style="background-color: whitesmoke; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(204, 204, 204); font-size: 12px; line-height: 1.4; padding: 1px 3px;">merge(A', B)</code> instead. Things would get further complicated if I want to commit multiple patches on my own branch and send in for review.</li>
</ul>
<div style="background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
All of the above mentioned problems are considered in lintstack. I invite you to checkout the <a href="https://github.com/openstack/nova/tree/master/tools" style="color: #3b73af; text-decoration: none;">code on github</a> to see the details.</div>
<h3 id="markdown-header-performance" style="background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 16px; line-height: 1.5625; margin: 20px 0px 5px;">
Performance</h3>
<div style="background-color: white; color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; margin-top: 10px;">
lintstack has been added to OpenStack Nova since late Folsom cycle. It's been deployed at the CI gate and integrated with the code review system gerrit at https://review.openstack.org. You can check the recent test results at <a href="https://jenkins.openstack.org/job/gate-nova-pylint/" style="color: #3b73af; text-decoration: none;">Jenkins</a>. During the entire Nova Grizzly development cycle, where 1877 change sets are committed, 127898 lines added, 79370 removed, only 18 errors reported from lintstatck are false positive cases, most of them are due to pylint unable to determine the call signatures of external libraries. On the other hand, lintstack almost finds real bugs (true positives) every day.</div>
<br />Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-2528454834564901150.post-30176779059973882622013-03-13T14:27:00.004-07:002013-03-13T14:50:55.195-07:00ZooKeeper and OpenStack Nova<pre class="brush: csharp"><h1 style="-webkit-font-smoothing: antialiased; border: 0px; cursor: text; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 28px; margin: 0px 0px 10px; padding: 0px; position: relative; white-space: normal;">
How to use the ZooKeeper driver for ServiceGroup in OpenStack Nova</h1>
Written by Yun Mao and Alexey Roytman (<a href="https://github.com/maoy/writeups/blob/master/ZooKeeperInNova.md">source on github</a>)
<h2 style="-webkit-font-smoothing: antialiased; border-bottom-color: rgb(204, 204, 204); border-bottom-style: solid; border-width: 0px 0px 1px; cursor: text; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 24px; margin: 0px 0px 10px; padding: 0px; position: relative; white-space: normal;">
<a class="anchor" href="https://github.com/maoy/writeups/blob/master/ZooKeeperInNova.md#introduction" name="introduction" style="border: 0px; bottom: 0px; color: #4183c4; cursor: pointer; display: block; left: 0px; margin: 0px 0px 0px -30px; padding: 0px 0px 0px 30px; position: absolute; text-decoration: none; top: 0px;"></a>Introduction</h2>
<div style="border: 0px; color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 14px; line-height: 22px; margin-bottom: 15px; padding: 0px; white-space: normal;">
If you are reading this page, I assume you have basic knowledge of what OpenStack is. But in case you don't know, OpenStack is a suite of software to turn your data center into a dynamic API driven cloud computing environment. Nova is the project name for OpenStack Compute, one of the key parts of an OpenStack. It manages all your compute nodes, where virtual machines (VMs) are hosted.</div>
<h2 style="-webkit-font-smoothing: antialiased; border-bottom-color: rgb(204, 204, 204); border-bottom-style: solid; border-width: 0px 0px 1px; cursor: text; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 24px; margin: 20px 0px 10px; padding: 0px; position: relative; white-space: normal;">
<a class="anchor" href="https://github.com/maoy/writeups/blob/master/ZooKeeperInNova.md#servicegroup-apis" name="servicegroup-apis" style="border: 0px; bottom: 0px; color: #4183c4; cursor: pointer; display: block; left: 0px; margin: 0px 0px 0px -30px; padding: 0px 0px 0px 30px; position: absolute; text-decoration: none; top: 0px;"></a>ServiceGroup APIs</h2>
<div style="border: 0px; color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 14px; line-height: 22px; margin-bottom: 15px; padding: 0px; white-space: normal;">
To effectively manage and utilize compute nodes, Nova needs to know the status of them. For example, when a user launches a new VM, the Nova scheduler should send the request to a <em style="border: 0px; margin: 0px; padding: 0px;">live</em> node (with enough capacity too, of course). Starting from the (upcoming) Grizzly release, Nova queries the ServiceGroup API to get the node liveness information.</div>
<div style="border: 0px; color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 14px; line-height: 22px; margin-bottom: 15px; margin-top: 15px; padding: 0px; white-space: normal;">
Here is roughly how the ServiceGroup API works. When a compute worker (running the <code style="background-color: #f8f8f8; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(234, 234, 234); font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 12px; margin: 0px 2px; padding: 0px 5px; white-space: nowrap;">nova-compute</code> daemon) starts, it calls the<code style="background-color: #f8f8f8; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(234, 234, 234); font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 12px; margin: 0px 2px; padding: 0px 5px; white-space: nowrap;">join</code> API to join the compute group, so that whoever is interested in the information (e.g. the scheduler) will be able to query the group membership (by call <code style="background-color: #f8f8f8; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(234, 234, 234); font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 12px; margin: 0px 2px; padding: 0px 5px; white-space: nowrap;">get_all</code> or <code style="background-color: #f8f8f8; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(234, 234, 234); font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 12px; margin: 0px 2px; padding: 0px 5px; white-space: nowrap;">get_one</code>), or the status of a particular node via the <code style="background-color: #f8f8f8; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(234, 234, 234); font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 12px; margin: 0px 2px; padding: 0px 5px; white-space: nowrap;">service_is_up</code> ServiceGroup API call. Under the hood, the ServiceGroup client driver will automatically update the compute worker status. If a certain node should explicitly be removed from the ServiceGroup, it can be done by the <code style="background-color: #f8f8f8; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(234, 234, 234); font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 12px; margin: 0px 2px; padding: 0px 5px; white-space: nowrap;">leave</code> call.</div>
<h2 style="-webkit-font-smoothing: antialiased; border-bottom-color: rgb(204, 204, 204); border-bottom-style: solid; border-width: 0px 0px 1px; cursor: text; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 24px; margin: 20px 0px 10px; padding: 0px; position: relative; white-space: normal;">
<a class="anchor" href="https://github.com/maoy/writeups/blob/master/ZooKeeperInNova.md#servicegroup-drivers" name="servicegroup-drivers" style="border: 0px; bottom: 0px; color: #4183c4; cursor: pointer; display: block; left: 0px; margin: 0px 0px 0px -30px; padding: 0px 0px 0px 30px; position: absolute; text-decoration: none; top: 0px;"></a>ServiceGroup Drivers</h2>
<div style="border: 0px; color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 14px; line-height: 22px; margin-bottom: 15px; padding: 0px; white-space: normal;">
Currently there are two drivers implemented and merged in master: database and ZooKeeper. More drivers are either in review (such as the memcached one), or in development.</div>
<h3 style="-webkit-font-smoothing: antialiased; border: 0px; color: #333333; cursor: text; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 18px; margin: 20px 0px 10px; padding: 0px; position: relative; white-space: normal;">
<a class="anchor" href="https://github.com/maoy/writeups/blob/master/ZooKeeperInNova.md#database-servicegroup-driver" name="database-servicegroup-driver" style="border: 0px; bottom: 0px; color: #4183c4; cursor: pointer; display: block; left: 0px; margin: 0px 0px 0px -30px; padding: 0px 0px 0px 30px; position: absolute; text-decoration: none; top: 0px;"></a>Database ServiceGroup driver</h3>
<div style="border: 0px; color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 14px; line-height: 22px; margin-bottom: 15px; padding: 0px; white-space: normal;">
The database driver is how Nova has been tracking node liveness since the first release, and it's the default driver. In a compute worker, it periodically sends a db update command to MySQL, saying "I'm OK" with a timestamp. A pre-defined timeout (<code style="background-color: #f8f8f8; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(234, 234, 234); font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 12px; margin: 0px 2px; padding: 0px 5px; white-space: nowrap;">service_down_time</code>) is used to determine if a node is dead. The driver has two limitations, which may or may not be an issue for you, depending on your setup. First, the more compute worker nodes you have, the more pressure you put on the database. Second, the timeout is by default 60 seconds. So, it might take a while to detect node failures. Of course you could try to reduce the timeout value, but you would also need to make the DB update more frequently, which again increases the DB workload.</div>
<div style="border: 0px; color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 14px; line-height: 22px; margin-bottom: 15px; margin-top: 15px; padding: 0px; white-space: normal;">
The fundamental issue is, the data to describe whether the node is alive is "transient" --- it's basically useless after a couple of seconds. Other data in the database, such as the entries to describe which users own what VMs, are persistent. But they are treated the same way since they are stored in the same database. This is also part of the motivation where we started the ServiceGroup abstraction so that we have a chance to treat them separately.</div>
<h3 style="-webkit-font-smoothing: antialiased; border: 0px; color: #333333; cursor: text; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 18px; margin: 20px 0px 10px; padding: 0px; position: relative; white-space: normal;">
<a class="anchor" href="https://github.com/maoy/writeups/blob/master/ZooKeeperInNova.md#zookeeper-servicegroup-driver" name="zookeeper-servicegroup-driver" style="border: 0px; bottom: 0px; color: #4183c4; cursor: pointer; display: block; left: 0px; margin: 0px 0px 0px -30px; padding: 0px 0px 0px 30px; position: absolute; text-decoration: none; top: 0px;"></a>ZooKeeper ServiceGroup driver</h3>
<h4 style="-webkit-font-smoothing: antialiased; border: 0px; color: #333333; cursor: text; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 16px; margin: 20px 0px 10px; padding: 0px; position: relative; white-space: normal;">
<a class="anchor" href="https://github.com/maoy/writeups/blob/master/ZooKeeperInNova.md#how-it-works" name="how-it-works" style="border: 0px; bottom: 0px; color: #4183c4; cursor: pointer; display: block; left: 0px; margin: 0px 0px 0px -30px; padding: 0px 0px 0px 30px; position: absolute; text-decoration: none; top: 0px;"></a>How it works</h4>
<div style="border: 0px; color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 14px; line-height: 22px; margin-bottom: 15px; padding: 0px; white-space: normal;">
The ZooKeeper ServiceGroup driver works by using ZooKeeper ephemeral nodes. ZooKeeper in cotrary to databases is a distributed system, and its load is divided among several servers.At a compute worker node, after establishing a ZooKeeper sesion, it creates an ephemeral znode in the group directory. Ephemeral znodes has the same lifespan as the session. If the worker node or the<code style="background-color: #f8f8f8; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(234, 234, 234); font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 12px; margin: 0px 2px; padding: 0px 5px; white-space: nowrap;">nova-compute</code> daemon crashes, or there is a network partition between the worker and the ZooKeeper server quorums, then the ephemeral znodes are removed automatically. The driver gets the group membership by doing a "ls" in group directory.</div>
<h4 style="-webkit-font-smoothing: antialiased; border: 0px; color: #333333; cursor: text; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 16px; margin: 20px 0px 10px; padding: 0px; position: relative; white-space: normal;">
<a class="anchor" href="https://github.com/maoy/writeups/blob/master/ZooKeeperInNova.md#installation-and-configuration" name="installation-and-configuration" style="border: 0px; bottom: 0px; color: #4183c4; cursor: pointer; display: block; left: 0px; margin: 0px 0px 0px -30px; padding: 0px 0px 0px 30px; position: absolute; text-decoration: none; top: 0px;"></a>Installation and configuration</h4>
<div style="border: 0px; color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 14px; line-height: 22px; margin-bottom: 15px; padding: 0px; white-space: normal;">
To use the ZooKeeper driver, of course you need to have both ZooKeeper servers and client libraries installed. Setting up ZooKeeper servers is outside the scope of this article. For the rest of the article, let's assume you have them installed, and their addresses / ports are 192.168.0.1:2181, 192.168.0.2:2181, 192.168.0.3:2181.</div>
<div style="border: 0px; color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 14px; line-height: 22px; margin-bottom: 15px; margin-top: 15px; padding: 0px; white-space: normal;">
To use ZooKeeper, you'll need two client-side Python libraries on every nova node. Here is the command to install then in Ubuntu:</div>
<pre style="background-color: #f8f8f8; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(204, 204, 204); color: #333333; font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 13px; line-height: 19px; margin-bottom: 15px; margin-top: 15px; overflow: auto; padding: 6px 10px;"><code style="background-color: transparent; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: none; font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 12px; margin: 0px; padding: 0px;">sudo apt-get install python-zookeeper python-pip
sudo pip install evzookeeper
</code></pre>
<div style="border: 0px; color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 14px; line-height: 22px; margin-bottom: 15px; margin-top: 15px; padding: 0px; white-space: normal;">
<code style="background-color: #f8f8f8; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(234, 234, 234); font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 12px; margin: 0px 2px; padding: 0px 5px; white-space: nowrap;">python-zookeeper</code> is the official ZooKeeper Python binding. <a href="https://github.com/maoy/python-evzookeeper" style="border: 0px; color: #4183c4; margin: 0px; padding: 0px; text-decoration: none;">evzookeeper</a> is the library to make the official binding work with the eventlet threading model.</div>
<div style="border: 0px; color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 14px; line-height: 22px; margin-bottom: 15px; margin-top: 15px; padding: 0px; white-space: normal;">
After installation, make sure you have the following configuration snippet at the end of <code style="background-color: #f8f8f8; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(234, 234, 234); font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 12px; margin: 0px 2px; padding: 0px 5px; white-space: nowrap;">/etc/nova/nova.conf</code> on every node:</div>
<pre style="background-color: #f8f8f8; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(204, 204, 204); color: #333333; font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 13px; line-height: 19px; margin-bottom: 15px; margin-top: 15px; overflow: auto; padding: 6px 10px;"><code style="background-color: transparent; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: none; font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 12px; margin: 0px; padding: 0px;">servicegroup_driver="zk"
[zookeeper]
address="192.168.0.1:2181,192.168.0.2:2181,192.168.0.3:2181"
</code></pre>
<div style="border: 0px; color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 14px; line-height: 22px; margin-top: 15px; padding: 0px; white-space: normal;">
That's it! After you start each Nova components, you can try to use <code style="background-color: #f8f8f8; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(234, 234, 234); font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 12px; margin: 0px 2px; padding: 0px 5px; white-space: nowrap;">nova-manage service list</code> to check the liveness of each compute node.</div>
</pre>
Unknownnoreply@blogger.com9