http://lukas.krejci.pw/Reinventing The Wheel2014-11-24T23:25:00+01:00http://lukas.krejci.pw/posts/2014/03/26/my-git-aliases/My git aliases2014-11-24T23:25:00+01:002014-03-26T00:00:00+00:00
Just a short note about the useful git aliases I’ve came up with during
the course of my daily work with git and github.
Although it’s been done by different people several times, I also came up
with some shortcuts to work with github’s pull requests from the
commandline. I tend to use github’s website only for commenting on the pull
requests but I like to do the rest locally in my repository (including the
merge of the pull request). Checkout this
gist where I keep my aliases.
...
<div class="paragraph">
<p>Just a short note about the useful git aliases I’ve came up with during
the course of my daily work with git and github.</p>
</div>
<div class="paragraph">
<p>Although it’s been done by different people several times, I also came up
with some shortcuts to work with github’s pull requests from the
commandline. I tend to use github’s website only for commenting on the pull
requests but I like to do the rest locally in my repository (including the
merge of the pull request). Checkout this
<a href="https://gist.github.com/metlos/9368527">gist</a> where I keep my aliases.</p>
</div>
http://lukas.krejci.pw/posts/2013/09/05/phantom-reads-and-data-paging/Phantom reads and data paging2014-11-24T23:25:00+01:002013-09-05T00:00:00+00:00
Paging through the results is easy, right?
The client only needs to supply the number of rows to skip and the
maximum number of rows it wants returned (aka the page number and the
page size). The server then returns the data along with the information
about the total number of results available. Et voila you have all the
information you need. The number of rows to skip together with the page
size give you the information about what page you’re showing and the
page size with the total number of rows gives you the total number of
pages available. Nothing too difficult or complex.
But there’s a catch....
<div class="paragraph">
<p>Paging through the results is easy, right?</p>
</div>
<div class="paragraph">
<p>The client only needs to supply the number of rows to skip and the
maximum number of rows it wants returned (aka the page number and the
page size). The server then returns the data along with the information
about the total number of results available. Et voila you have all the
information you need. The number of rows to skip together with the page
size give you the information about what page you’re showing and the
page size with the total number of rows gives you the total number of
pages available. Nothing too difficult or complex.</p>
</div>
<div class="paragraph">
<p>But there’s a catch. On the server, one needs to perform (at least) two
queries - one query to get the data for the requested page and the
second query to fetch the total number of rows. Now most of the
databases set the default transaction isolation level to READ_COMMITTED
and for very good reasons. But this transaction isolation level allows
for phantom reads, i.e. 2 queries in the same transaction might "see"
different number of rows of data, if another transaction committed and
added or deleted rows that would be returned by the queries. So, it may
happen that you will:<br></p>
</div>
<div class="ulist">
<ul>
<li>
<p>return "rows 5 to 10 out of 2 total",</p>
</li>
<li>
<p>say "there are no available results on the first page, while the total
number of rows is 5",</p>
</li>
<li>
<p>etc.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>All that info acquired within one transaction.</p>
</div>
<div class="paragraph">
<p>What can you do about such situations? The obvious solution is to just
admit that these things can happen ;) Another option is to try and
detect if such situation might have occured and re-try.</p>
</div>
<div class="paragraph">
<p>I’ve come up with the following rules for consistency of the results:</p>
</div>
<div class="paragraph">
<p><strong><code>N</code></strong> is the actual number of elements on the page, <strong><code>P</code></strong> is the maximum
number of elements on the page (i.e. the page size), <strong><code>I</code></strong> is the number
of rows to skip and <strong><code>T</code></strong> is total number of results.</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>T < I && N == 0</code>. This means we’re trying to show a page that is past
the total number of results. We therefore expect the collection to be
empty.</p>
</li>
<li>
<p><code>T - I > P && N == P</code>. If we are are not showing the last page, the
number of elements in the collection should be equal to the page size.</p>
</li>
<li>
<p><code>T - I ⇐ P && N == T - I</code>. If showing the last page, the collection
should have all the remaining elements in it.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>These are kind of obvious assumptions but phantom read can easily break
them and therefore one should be checking them if one wants to be
serious about returning meaningful results to the user.</p>
</div>
<div class="paragraph">
<p>So while paging is simple in principle, there are a couple of
interesting corner cases that one needs to handle if one reads data out
of a dynamic data set. It took us a good couple of years in
<a href="http://www.jboss.org/rhq">RHQ</a> to get to the bottom of this but hopefully
now our paging is more robust than it was before.</p>
</div>
http://lukas.krejci.pw/posts/2012/09/27/scripting-news-in-rhq-4-5-0/Scripting News in RHQ 4.5.02014-11-24T23:25:00+01:002012-09-27T00:00:00+00:00
RHQ 4.5.0
(which we released today) contains a great deal of scripting
enhancements that I think are worth talking about in more detail. In my
eyes, the changes make the scripting in RHQ ready for serious use.
CommonJS support
This, I think, is huge. In the previous versions of RHQ, the only way of
reusing functionality from another script was to use the exec -f
command in the interactive CLI shell (in another words, this was NOT
available in the batch mode, which is how majority of people are using
the CLI). So if you needed to implement something bigger and needed to
split your code in several files (as...
<div class="paragraph">
<p><a href="https://docs.jboss.org/author/display/RHQ/Release+Notes+4.5.0">RHQ 4.5.0</a>
(which we released today) contains a great deal of scripting
enhancements that I think are worth talking about in more detail. In my
eyes, the changes make the scripting in RHQ ready for serious use.</p>
</div>
<div class="sect3">
<h4 id="commonjs-support">
<a class="anchor" href="#commonjs-support"></a>CommonJS support</h4>
<div class="paragraph">
<p>This, I think, is huge. In the previous versions of RHQ, the only way of
reusing functionality from another script was to use the <code>exec -f</code>
command in the interactive CLI shell (in another words, this was NOT
available in the batch mode, which is how majority of people are using
the CLI). So if you needed to implement something bigger and needed to
split your code in several files (as any sane person would do), you only
had 1 option - before executing the "script", you needed to concatenate
all the scripts together.</p>
</div>
<div class="paragraph">
<p>This sucked big time and we knew it ;) But we didn’t want to just add
functionality to "include files" - that would be too easy ;) At the same
time it wouldn’t solve the problem, really. The problem with just
"including" the files into the current "scope" of the script is that
this would mean that each and every variable or function in those files
would have to be uniquely named because javascript lacks any sort of
namespace/package resolution. Fortunately, the
<a href="http://commonjs.org">CommonJS</a> spec solves this problem.</p>
</div>
<div class="paragraph">
<p>Here’s how you use a module. Notice that you assign the module to a
variable and that’s how you prevent the "pollution" of your scope. The
loaded module can have methods and variables with the same name as your
script and they won’t influence each other:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="javascript"><table class="CodeRay"><tr>
<td class="line-numbers"><pre>1
2</pre></td>
<td class="code"><pre><span class="keyword">var</span> myfuncs = require(<span class="string"><span class="delimiter">"</span><span class="content">modules:/myfuncs</span><span class="delimiter">"</span></span>);
myfuncs.helloworld();</pre></td>
</tr></table></code></pre>
</div>
</div>
<div class="paragraph">
<p>You may wonder what that <code>"modules:/myfuncs"</code> identifier means. It is an
URI that the underlying library uses to locate the script to load. This
"sourcing" mechanism is pluggable and I will talk about it more in the
following chapter. To see some examples of the modules, you can look at
the
<a href="http://git.fedorahosted.org/cgit/rhq/rhq.git/tree/modules/enterprise/remoting/cli/src/main/samples/modules"><code>samples/modules</code></a>
directory of your CLI deployment and you can also read some
documentation about this on
<a href="https://docs.jboss.org/author/display/RHQ/Script+Modularity+-+Javascript">our
wiki</a>.</p>
</div>
</div>
<div class="sect3">
<h4 id="locating-the-scripts">
<a class="anchor" href="#locating-the-scripts"></a>Locating the scripts</h4>
<div class="paragraph">
<p>With the ability to load the scripts there comes the problem of locating
them. For the standalone CLI, the obvious location for them is the
filesystem, but what about alert notification scripts on the RHQ server?
These scripts are stored in RHQ repositories which don’t have a
filesystem location. The solution is not to tie the scripts to the
filesystem but have a generic way of locating them using URIs and a
pluggable way of resolving those URIs and loading the scripts from
various locations. This means that you can for example load the script
from an RHQ repository in your standalone CLI installation, or to define
1 central location for your local CLI scripts and use the "modules" URIs
to refer to them. Or you can easily implement your own "source provider"
and for example load the scripts from your internal git repo or ftp or
whatnot. RHQ comes with a small set of predefined source providers,
documented
<a href="https://docs.jboss.org/author/display/RHQ/Script+Sources">here</a>.</p>
</div>
<div class="paragraph">
<p>With this ability at hand, you can make an RHQ repository a central
place for your scripts that you will then be able to use universally -
both in the standalone CLI installations and also in the alert
notification scripts.</p>
</div>
</div>
<div class="sect3">
<h4 id="python">
<a class="anchor" href="#python"></a>Python</h4>
<div class="paragraph">
<p>In previous versions, our scripting was tied to Javascript. Thanks to
quite a bit of refactoring, the RHQ scripting integration is now
language independent and language support is pluggable (see my previous
<a href="http://metlos.wordpress.com/2012/07/12/rhq-speaks-python/">post</a> where I
detail how this was done in case of Python).</p>
</div>
<div class="paragraph">
<p>What this all means is that you can now write your CLI scripts in Python
and still use the same API as you were able to use before from
Javascript only. I.e. you will find the <code>ResourceManager</code>,
<code>AlertManager</code> and all the other predefined variables that define the
RHQ API available in Python, too. The only thing that this initial
implementation doesn’t support is code-completion in the interactive CLI
shell.</p>
</div>
<div class="paragraph">
<p>Last but not least, the ability load the scripts from various locations
is available in Python, too, using an automatically installed
<a href="http://www.python.org/dev/peps/pep-0302/">path_hook</a>. You can read about
how to use it on our
<a href="https://docs.jboss.org/author/display/RHQ/Script+Modularity+-+Python">wiki</a>.
This also means that you can now write your alert notification scripts
in Python, too.</p>
</div>
<div class="paragraph">
<p>When running an alert notification script (i.e. an alert notification of
the type "CLI Script"), the language of the script is determined from
the script file extension - ".py" for python and ".js" for javascript.
When you start the CLI shell, you pass your language of choice using the
<code>--language</code> commandline parameter - "javascript" or "python" are the
obvious possible values for it.</p>
</div>
</div>
<div class="sect3">
<h4 id="conclusion">
<a class="anchor" href="#conclusion"></a>Conclusion</h4>
<div class="paragraph">
<p>In my opinion, these changes are great and will allow our users to
really start building useful tools using our scripting support. If you
feel like you’ve come up with a script module you would like to share
with the RHQ community, why don’t you just send a pull request to our
github <a href="https://github.com/rhq-project/samples">repo</a> with sample scripts?</p>
</div>
</div>
http://lukas.krejci.pw/posts/2012/09/14/the-dark-powers-of-powermock/The Dark Powers of PowerMock2014-11-24T23:25:00+01:002012-09-14T00:00:00+00:00
Recently, we’ve started using Mockito
and PowerMock in our testing. I
won’t explain mocking and why or why not you should use it, but I want
to share my experience with using PowerMock.
PowerMock comes with a very strong promise: "PowerMock uses a custom
classloader and bytecode manipulation to enable mocking of static
methods, constructors, final classes and methods, private methods,
removal of static initializers and more."
That is seriously cool, right? I thought so, too, but I stumbled upon
several problems the very first time I tried to use it. Frankly, those
problems, as always, stemmed from my lack of experience with the tool,
but hey - everyone’s a novice...
<div class="paragraph">
<p>Recently, we’ve started using <a href="http://code.google.com/p/mockito/">Mockito</a>
and <a href="http://code.google.com/p/powermock/">PowerMock</a> in our testing. I
won’t explain mocking and why or why not you should use it, but I want
to share my experience with using PowerMock.</p>
</div>
<div class="paragraph">
<p><code>PowerMock</code> comes with a very strong promise: "PowerMock uses a custom
classloader and bytecode manipulation to enable mocking of static
methods, constructors, final classes and methods, private methods,
removal of static initializers and more."</p>
</div>
<div class="paragraph">
<p>That is seriously cool, right? I thought so, too, but I stumbled upon
several problems the very first time I tried to use it. Frankly, those
problems, as always, stemmed from my lack of experience with the tool,
but hey - everyone’s a novice at first. Let me share my experience with
you.</p>
</div>
<div class="sect3">
<h4 id="the-problem">
<a class="anchor" href="#the-problem"></a>The Problem</h4>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="java"><table class="CodeRay"><tr>
<td class="line-numbers"><pre>1
2
3
4
5
6
7</pre></td>
<td class="code"><pre><span class="directive">public</span> <span class="type">class</span> <span class="class">ClassUnderTest</span> {
<span class="directive">public</span> <span class="predefined-type">InputStream</span> method(<span class="type">boolean</span> param, <span class="predefined-type">URI</span> uri) <span class="directive">throws</span> <span class="exception">Exception</span> {
<span class="predefined-type">String</span> scheme = param ? <span class="string"><span class="delimiter">"</span><span class="content">https</span><span class="delimiter">"</span></span> : <span class="string"><span class="delimiter">"</span><span class="content">http</span><span class="delimiter">"</span></span>;
<span class="predefined-type">URI</span> replacedUri = <span class="keyword">new</span> <span class="predefined-type">URI</span>(scheme, uri.getAuthority(), uri.getPath(), uri.getQuery(), uri.getFragment());
<span class="keyword">return</span> replacedUri.toURL().openStream();
}
}</pre></td>
</tr></table></code></pre>
</div>
</div>
<div class="paragraph">
<p>The above fabricated example expresses the essence of the testing
challenge I faced (the real class was
<a href="http://git.fedorahosted.org/cgit/rhq/rhq.git/tree/modules/enterprise/remoting/client-api/src/main/java/org/rhq/enterprise/clientapi/RhqDownloadsScriptSourceProvider.java">this</a>.)
The method I wanted to test obtains an URI and transforms it based on
some parameters. Then it tries to open a stream on the URI so that the
caller can download the contents.</p>
</div>
<div class="paragraph">
<p>Because the URI that the method tries to download from is by design
either http or https URL, it is kind of hard to test without actually
standing up a HTTP server to serve the file during the test. This is of
course not impossible and possibly would not be that hard, but I thought
<code>PowerMock</code> can come here to the rescue. I should be able to mock those
calls out in my tests.</p>
</div>
</div>
<div class="sect3">
<h4 id="attempt-1-mocking-system-classes">
<a class="anchor" href="#attempt-1-mocking-system-classes"></a>Attempt #1 - mocking system classes</h4>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="java"><table class="CodeRay"><tr>
<td class="line-numbers"><pre>1
2
3
4
5
6
7
8
9
<strong>10</strong>
11
12
13
14
15
16
17
18
19
<strong>20</strong></pre></td>
<td class="code"><pre><span class="annotation">@Test</span>
<span class="annotation">@PrepareForTest</span>(ClassUnderTest.class)
<span class="directive">public</span> <span class="type">class</span> <span class="class">MyTest</span> {
<span class="annotation">@ObjectFactory</span>
<span class="directive">public</span> IObjectFactory getObjectFactory() {
<span class="keyword">return</span> <span class="keyword">new</span> PowerMockObjectFactory();
}
<span class="directive">public</span> <span class="type">void</span> testMethod() <span class="directive">throws</span> <span class="exception">Exception</span> {
<span class="predefined-type">URI</span> uriMock = PowerMockito.mock(<span class="predefined-type">URI</span>.class);
<span class="predefined-type">URL</span> urlMock = PowerMockito.mock(<span class="predefined-type">URL</span>.class);
PowerMockito.whenNew(<span class="predefined-type">URI</span>.class).withArguments(<span class="string"><span class="delimiter">"</span><span class="content">http</span><span class="delimiter">"</span></span>, <span class="string"><span class="delimiter">"</span><span class="content">localhost</span><span class="delimiter">"</span></span>, <span class="predefined-constant">null</span>, <span class="predefined-constant">null</span>, <span class="predefined-constant">null</span>).thenReturn(uriMock);
Mockito.when(uriMock.toURL()).thenReturn(urlMock);
Mockito.when(urlMock.openStream()).thenReturn(<span class="keyword">new</span> <span class="predefined-type">FileInputStream</span>(<span class="keyword">new</span> <span class="predefined-type">File</span>(<span class="string"><span class="delimiter">"</span><span class="content">.</span><span class="delimiter">"</span></span>, <span class="string"><span class="delimiter">"</span><span class="content">existing.file</span><span class="delimiter">"</span></span>)));
ClassUnderTest testObject = <span class="keyword">new</span> ClassUnderTest();
testObject.method(<span class="predefined-constant">false</span>, <span class="keyword">new</span> <span class="predefined-type">URI</span>(<span class="string"><span class="delimiter">"</span><span class="content">blah://localhost</span><span class="delimiter">"</span></span>));
}
}</pre></td>
</tr></table></code></pre>
</div>
</div>
<div class="paragraph">
<p>This should be fairly easy to understand for everyone that used some
mocking framework. I’m creating two mocks: one for URI and one for URL
classes. Then I’m using PowerMock to capture the construction of a new
URI (see the code of the ClassUnderTest) and returning my <code>uriMock</code>. The
uriMock is set up to return the <code>urlMock</code> when its <code>toURL()</code> method is
called. When the <code>openStream()</code> method is called on my <code>urlMock</code>, I’m
returning an input stream of a local file.</p>
</div>
<div class="paragraph">
<p>Nice and easy, right? Except it doesn’t work. I get the following
stacktrace as soon as I try to mock the URI class:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code>org.mockito.exceptions.base.MockitoException: Mockito cannot mock this class: class replica.java.net.URI$$PowerMock0 Mockito can only mock visible & non-final classes.</code></pre>
</div>
</div>
<div class="paragraph">
<p>After a bit of googling, the cause is apparent - PowerMock cannot mock
the system classes (unless PowerMock java agent is used). Ok, let’s try
another approach, this time trying to avoid using mocks.</p>
</div>
</div>
<div class="sect3">
<h4 id="attempt-2-powermockito-whennew-url-class">
<a class="anchor" href="#attempt-2-powermockito-whennew-url-class"></a>Attempt #2 - PowerMockito.whenNew(URL.class)</h4>
<div class="paragraph">
<p>The idea behind this attempt is that <code>PowerMockito</code> can capture and
override constructor calls. Because <code>URI.toURL()</code> constructs a new URL
instance with a single string argument, so we theoretically should be
able to intercept that?</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="java"><table class="CodeRay"><tr>
<td class="line-numbers"><pre>1
2
3
4
5
6
7
8</pre></td>
<td class="code"><pre><span class="directive">public</span> <span class="type">void</span> testMethod() <span class="directive">throws</span> <span class="exception">Exception</span> {
<span class="predefined-type">URL</span> realUrl = <span class="keyword">new</span> <span class="predefined-type">File</span>(<span class="string"><span class="delimiter">"</span><span class="content">.</span><span class="delimiter">"</span></span>, <span class="string"><span class="delimiter">"</span><span class="content">existing.file</span><span class="delimiter">"</span></span>).toURI().toURL();
PowerMockito.whenNew(<span class="predefined-type">URL</span>.class).withArguments(<span class="string"><span class="delimiter">"</span><span class="content">http://localhost</span><span class="delimiter">"</span></span>).thenReturn(realUrl);
ClassUnderTest testObject = <span class="keyword">new</span> ClassUnderTest();
testObject.method(<span class="predefined-constant">false</span>, <span class="keyword">new</span> <span class="predefined-type">URI</span>(<span class="string"><span class="delimiter">"</span><span class="content">blah://localhost</span><span class="delimiter">"</span></span>));
}</pre></td>
</tr></table></code></pre>
</div>
</div>
<div class="paragraph">
<p>As you might have guessed, this doesn’t work either. And frankly if it
did, I’d have some serious questions about how it could. The constructor
of <code>URL</code> is only called inside the <code>toURL()</code> of the <code>URI</code> which is a
system class that PowerMock can’t touch. So, the third attempt.</p>
</div>
</div>
<div class="sect3">
<h4 id="attempt-3-powermockito-whennew-uri-class">
<a class="anchor" href="#attempt-3-powermockito-whennew-uri-class"></a>Attempt #3 - PowerMockito.whenNew(URI.class)</h4>
<div class="paragraph">
<p>What is the difference between this one and the previous attempt? Well,
it took me a while to decipher the
<a href="http://powermock.googlecode.com/svn/docs/powermock-1.3.5/apidocs/org/powermock/core/classloader/annotations/PrepareForTest.html">javadoc</a>
for the <code>@PrepareForTest</code> annotation, but it boils down to this. If you
need to use the <code>PowerMockito.whenNew</code> method, you need to tell
PowerMock to do bytecode manipulation on the class that (in some method)
<strong>directly calls</strong> given constructor. This is kinda understandable when
you know what PowerMock is doing - it will actually change the byte code
of the "prepared" class so that any constructor calls (and other things)
are checking for the rules defined using <code>whenNew</code> and other methods.
You realize this for real when you try to debug the class under test
(that has been prepared by power mock) - you can no longer be sure that
what you see in the code is actually what is happening, because the
bytecode of the class no longer exactly corresponds to what you see in
the source code.</p>
</div>
<div class="paragraph">
<p>So to sum it up, here’s the code that works:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="java"><table class="CodeRay"><tr>
<td class="line-numbers"><pre>1
2
3
4
5
6
7
8
9
<strong>10</strong>
11
12
13
14
15
16</pre></td>
<td class="code"><pre><span class="annotation">@Test</span>
<span class="annotation">@PrepareForTest</span>(ClassUnderTest.class)
<span class="directive">public</span> <span class="type">class</span> <span class="class">MyTest</span> {
<span class="annotation">@ObjectFactory</span>
<span class="directive">public</span> IObjectFactory getObjectFactory() {
<span class="keyword">return</span> <span class="keyword">new</span> PowerMockObjectFactory();
}
<span class="directive">public</span> <span class="type">void</span> testMethod() <span class="directive">throws</span> <span class="exception">Exception</span> {
<span class="predefined-type">URI</span> realUri = <span class="keyword">new</span> <span class="predefined-type">File</span>(<span class="string"><span class="delimiter">"</span><span class="content">.</span><span class="delimiter">"</span></span>, <span class="string"><span class="delimiter">"</span><span class="content">existing.file</span><span class="delimiter">"</span></span>).toURI();
PowerMockito.whenNew(<span class="predefined-type">URI</span>.class).withArguments(<span class="string"><span class="delimiter">"</span><span class="content">http</span><span class="delimiter">"</span></span>, <span class="string"><span class="delimiter">"</span><span class="content">localhost</span><span class="delimiter">"</span></span>, <span class="predefined-constant">null</span>, <span class="predefined-constant">null</span>, <span class="predefined-constant">null</span>).thenReturn(realUri);
ClassUnderTest testObject = <span class="keyword">new</span> ClassUnderTest();
testObject.method(<span class="predefined-constant">false</span>, <span class="keyword">new</span> <span class="predefined-type">URI</span>(<span class="string"><span class="delimiter">"</span><span class="content">blah://localhost</span><span class="delimiter">"</span></span>));
}
}</pre></td>
</tr></table></code></pre>
</div>
</div>
<div class="paragraph">
<p>The constructor of the <code>URI</code> is intercepted and we return a "realUri",
i.e. a different instance of otherwise "normal" URI class. This works,
because exactly that constructor with those arguments is called in the
class under test that has been manipulated by PowerMock (as instructed
by the <code>@PrepareForTest</code> annotation). From that point on, we don’t need
any special behavior on either the <code>URI</code> or <code>URL</code> classes and so the
code can stay untouched.</p>
</div>
</div>
<div class="sect3">
<h4 id="conclusion">
<a class="anchor" href="#conclusion"></a>Conclusion</h4>
<div class="paragraph">
<p>The conclusion is basically the famous 4 letters - RTFM :-) I just
wanted to detail my journey through the dark corners of the PowerMock
forest just in case some of you were as confused as I was when I first
entered it.</p>
</div>
</div>
http://lukas.krejci.pw/posts/2012/07/12/rhq-speaks-python/RHQ speaks Python2014-11-24T23:25:00+01:002012-07-12T00:00:00+00:00
In the past few weeks I was quite busy refactoring RHQ’s CLI and
scripting integration. Funnily enough it all started because we wanted
to add the support for CommonJS modules to our javascript interface.
During the course of the refactoring, I found out that I’m actually
heading in the direction of completely separating the "language" support
from the rest of the RHQ, which then only speaks to it through the
Java’s standard scripting APIs which are language independent.
RHQ’s CLI was originally only implemented for and tightly coupled with
javascript for which the JRE has support by default. The problem we had
was that the version of Rhino (i.e....
<div class="paragraph">
<p>In the past few weeks I was quite busy refactoring RHQ’s CLI and
scripting integration. Funnily enough it all started because we wanted
to add the support for CommonJS modules to our javascript interface.
During the course of the refactoring, I found out that I’m actually
heading in the direction of completely separating the "language" support
from the rest of the RHQ, which then only speaks to it through the
Java’s standard scripting APIs which are language independent.</p>
</div>
<div class="paragraph">
<p>RHQ’s CLI was originally only implemented for and tightly coupled with
javascript for which the JRE has support by default. The problem we had
was that the version of Rhino (i.e. the Javascript implementation Java
uses) that is bundled with the JRE does not support CommonJS modules
while the newer versions do.</p>
</div>
<div class="paragraph">
<p>But this is about Python, right? So once I saw that we have a nice
little API that one can implement to add support for another language, I
thought why not try bringing another language to RHQ? The obvious choice
was Python - the most popular language among the ones that can integrate
with Java. So I grabbed Jython and started looking if would be possible
to do with it everything we needed to do to implement our API. And it
turned out it was - a mere 200 lines of Java code and RHQ can speak
Python :)</p>
</div>
<div class="paragraph">
<p>Let’s look at how the API we needed implement looked like:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="java"><table class="CodeRay"><tr>
<td class="line-numbers"><pre>1
2
3
4
5
6
7
8
9
<strong>10</strong>
11
12
13
14
15
16
17
18</pre></td>
<td class="code"><pre><span class="directive">public</span> <span class="type">class</span> <span class="class">PythonScriptEngineProvider</span> <span class="directive">implements</span> ScriptEngineProvider {
<span class="annotation">@Override</span>
<span class="directive">public</span> <span class="predefined-type">String</span> getSupportedLanguage() {
<span class="keyword">return</span> <span class="string"><span class="delimiter">"</span><span class="content">python</span><span class="delimiter">"</span></span>;
}
<span class="annotation">@Override</span>
<span class="directive">public</span> ScriptEngineInitializer getInitializer() {
<span class="keyword">return</span> <span class="keyword">new</span> PythonScriptEngineInitializer();
}
<span class="annotation">@Override</span>
<span class="directive">public</span> CodeCompletion getCodeCompletion() {
<span class="comment">// XXX are we gonna support code completion for multiple langs in the CLI?</span>
<span class="keyword">return</span> <span class="predefined-constant">null</span>;
}
}</pre></td>
</tr></table></code></pre>
</div>
</div>
<div class="paragraph">
<p>Now that’s quite trivial, isn’t it? :) Of course, this is the basic
interface which just delegates the real work to other classes. So let’s
look at the <code>ScriptEngineInitializer</code> - the class that really does the
all the important work:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="java"><table class="CodeRay"><tr>
<td class="line-numbers"><pre>1
2
3
4
5
6
7
8
9
<strong>10</strong>
11
12
13
14
15
16
17
18
19
<strong>20</strong>
21
22
23
24
25
26
27
28
29
<strong>30</strong>
31
32
33
34
35
36
37
38
39
<strong>40</strong>
41
42
43
44
45
46
47
48
49
<strong>50</strong>
51
52
53
54
55
56
57
58
59
<strong>60</strong>
61
62
63
64
65
66
67
68
69
<strong>70</strong>
71
72
73
74
75
76
77
78
79
<strong>80</strong>
81
82
83
84</pre></td>
<td class="code"><pre><span class="directive">public</span> <span class="type">class</span> <span class="class">PythonScriptEngineInitializer</span> <span class="directive">implements</span> ScriptEngineInitializer {
<span class="directive">private</span> <span class="directive">static</span> <span class="directive">final</span> Log LOG = LogFactory.getLog(PythonScriptEngineInitializer.class);
<span class="directive">static</span> {
<span class="predefined-type">Properties</span> props = <span class="keyword">new</span> <span class="predefined-type">Properties</span>();
props.put(<span class="string"><span class="delimiter">"</span><span class="content">python.packages.paths</span><span class="delimiter">"</span></span>, <span class="string"><span class="delimiter">"</span><span class="content">java.class.path,sun.boot.class.path</span><span class="delimiter">"</span></span>);
props.put(<span class="string"><span class="delimiter">"</span><span class="content">python.packages.directories</span><span class="delimiter">"</span></span>, <span class="string"><span class="delimiter">"</span><span class="content">java.ext.dirs</span><span class="delimiter">"</span></span>);
props.put(<span class="string"><span class="delimiter">"</span><span class="content">python.cachedir.skip</span><span class="delimiter">"</span></span>, <span class="predefined-constant">false</span>);
PythonInterpreter.initialize(<span class="predefined-type">System</span>.getProperties(), props, <span class="predefined-constant">null</span>);
}
<span class="directive">private</span> ScriptEngineManager engineManager = <span class="keyword">new</span> ScriptEngineManager();
<span class="annotation">@Override</span>
<span class="directive">public</span> ScriptEngine instantiate(<span class="predefined-type">Set</span> packages, <span class="predefined-type">PermissionCollection</span> permissions) <span class="directive">throws</span> ScriptException {
ScriptEngine eng = engineManager.getEngineByName(<span class="string"><span class="delimiter">"</span><span class="content">python</span><span class="delimiter">"</span></span>);
<span class="comment">//XXX this might not work perfectly in jython</span>
<span class="comment">//but we can't make it work perfectly either, so let's just</span>
<span class="comment">//keep our fingers crossed..</span>
<span class="comment">//http://www.jython.org/jythonbook/en/1.0/ModulesPackages.html#from-import-statements</span>
<span class="keyword">for</span> (<span class="predefined-type">String</span> pkg : packages) {
<span class="keyword">try</span> {
eng.eval(<span class="string"><span class="delimiter">"</span><span class="content">from </span><span class="delimiter">"</span></span> + pkg + <span class="string"><span class="delimiter">"</span><span class="content"> import *</span><span class="char">\n</span><span class="delimiter">"</span></span>);
} <span class="keyword">catch</span> (ScriptException e) {
<span class="comment">//well, let's just keep things going, this is not fatal...</span>
LOG.info(<span class="string"><span class="delimiter">"</span><span class="content">Python script engine could not pre-import members of package '</span><span class="delimiter">"</span></span> + pkg + <span class="string"><span class="delimiter">"</span><span class="content">'.</span><span class="delimiter">"</span></span>);
}
}
<span class="comment">//fingers crossed we can secure jython like this</span>
<span class="keyword">return</span> permissions == <span class="predefined-constant">null</span> ? eng : <span class="keyword">new</span> SandboxedScriptEngine(eng, permissions);
}
<span class="annotation">@Override</span>
<span class="directive">public</span> <span class="type">void</span> installScriptSourceProvider(ScriptEngine scriptEngine, ScriptSourceProvider provider) {
PySystemState sys = Py.getSystemState();
<span class="keyword">if</span> (sys != <span class="predefined-constant">null</span>) {
sys.path_hooks.append(<span class="keyword">new</span> PythonSourceProvider(provider));
}
}
<span class="annotation">@Override</span>
<span class="directive">public</span> <span class="predefined-type">Set</span> generateIndirectionMethods(<span class="predefined-type">String</span> boundObjectName, <span class="predefined-type">Set</span> overloadedMethods) {
<span class="keyword">if</span> (overloadedMethods == <span class="predefined-constant">null</span> || overloadedMethods.isEmpty()) {
<span class="keyword">return</span> <span class="predefined-type">Collections</span>.emptySet();
}
<span class="predefined-type">Set</span> argCnts = <span class="keyword">new</span> <span class="predefined-type">HashSet</span>();
<span class="keyword">for</span> (<span class="predefined-type">Method</span> m : overloadedMethods) {
argCnts.add(m.getParameterTypes().length);
}
<span class="predefined-type">String</span> methodName = overloadedMethods.iterator().next().getName();
<span class="predefined-type">StringBuilder</span> functionBody = <span class="keyword">new</span> <span class="predefined-type">StringBuilder</span>();
functionBody.append(<span class="string"><span class="delimiter">"</span><span class="content">def </span><span class="delimiter">"</span></span>).append(methodName).append(<span class="string"><span class="delimiter">"</span><span class="content">(*args, **kwargs):</span><span class="char">\n</span><span class="delimiter">"</span></span>);
functionBody.append(<span class="string"><span class="delimiter">"</span><span class="char">\t</span><span class="delimiter">"</span></span>).append(<span class="string"><span class="delimiter">"</span><span class="content">if len(kwargs) > 0:</span><span class="char">\n</span><span class="delimiter">"</span></span>);
functionBody.append(<span class="string"><span class="delimiter">"</span><span class="char">\t</span><span class="char">\t</span><span class="delimiter">"</span></span>).append(<span class="string"><span class="delimiter">"</span><span class="content">raise ValueError(</span><span class="char">\"</span><span class="content">Named arguments not supported for Java methods</span><span class="char">\"</span><span class="content">)</span><span class="char">\n</span><span class="delimiter">"</span></span>);
functionBody.append(<span class="string"><span class="delimiter">"</span><span class="char">\t</span><span class="delimiter">"</span></span>).append(<span class="string"><span class="delimiter">"</span><span class="content">argCnt = len(args)</span><span class="char">\n</span><span class="delimiter">"</span></span>);
<span class="keyword">for</span> (<span class="predefined-type">Integer</span> argCnt : argCnts) {
functionBody.append(<span class="string"><span class="delimiter">"</span><span class="char">\t</span><span class="delimiter">"</span></span>).append(<span class="string"><span class="delimiter">"</span><span class="content">if argCnt == </span><span class="delimiter">"</span></span>).append(argCnt).append(<span class="string"><span class="delimiter">"</span><span class="content">:</span><span class="char">\n</span><span class="delimiter">"</span></span>);
functionBody.append(<span class="string"><span class="delimiter">"</span><span class="char">\t</span><span class="char">\t</span><span class="content">return </span><span class="delimiter">"</span></span>).append(boundObjectName).append(<span class="string"><span class="delimiter">"</span><span class="content">.</span><span class="delimiter">"</span></span>).append(methodName).append(<span class="string"><span class="delimiter">"</span><span class="content">(</span><span class="delimiter">"</span></span>);
<span class="type">int</span> last = argCnt - <span class="integer">1</span>;
<span class="keyword">for</span> (<span class="type">int</span> i = <span class="integer">0</span>; i < argCnt; ++i) {
functionBody.append(<span class="string"><span class="delimiter">"</span><span class="content">args[</span><span class="delimiter">"</span></span>).append(i).append(<span class="string"><span class="delimiter">"</span><span class="content">]</span><span class="delimiter">"</span></span>);
<span class="keyword">if</span> (i < last) {
functionBody.append(<span class="string"><span class="delimiter">"</span><span class="content">, </span><span class="delimiter">"</span></span>);
}
}
functionBody.append(<span class="string"><span class="delimiter">"</span><span class="content">)</span><span class="char">\n</span><span class="delimiter">"</span></span>);
}
<span class="keyword">return</span> <span class="predefined-type">Collections</span>.singleton(functionBody.toString());
}
<span class="annotation">@Override</span>
<span class="directive">public</span> <span class="predefined-type">String</span> extractUserFriendlyErrorMessage(ScriptException e) {
<span class="keyword">return</span> e.getMessage();
}
}</pre></td>
</tr></table></code></pre>
</div>
</div>
<div class="paragraph">
<p>The most important task of the initializer is to instantiate the script
engine of the language it supports and intialize it - pre-import java
packages of RHQ’s classes and apply java security to the script engine.
The other tasks it has are to install a "script source provider" to the
engine (the script source provider is a class that is able to locate a
script "somewhere"), to extract a user-friendly error message from the
script exception and finally to generate "indirection methods" -
basically define top level functions that delegate to a method on
certain object. All these methods are there so that RHQ can correctly
set up the bindings that the scripts then can use to access and
manipulate RHQ data.</p>
</div>
<div class="paragraph">
<p>I won’t be listing the source of the class that integrates the source
providers with Python, you can take a look at it
<a href="http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=blob;f=modules/enterprise/scripting/python/src/main/java/org/rhq/scripting/python/PythonSourceProvider.java;hb=HEAD">here</a>.
But I’ll show you how it is possible in your local CLI session to import
a python script stored in the RHQ server in some repository:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="python"><table class="CodeRay"><tr>
<td class="line-numbers"><pre>1
2
3
4
5
6
7</pre></td>
<td class="code"><pre><span class="keyword">import</span> <span class="include">sys</span>
sys.path.append(<span class="string"><span class="delimiter">"</span><span class="content">__rhq__:rhq://repositories/my_repo/</span><span class="delimiter">"</span></span>)
<span class="keyword">import</span> <span class="include">my_script</span> <span class="keyword">as</span> foo
...</pre></td>
</tr></table></code></pre>
</div>
</div>
<div class="paragraph">
<p>RHQ has a <code>path_hook</code> in Python that looks for paths prefixed with
<code><em>rhq</em>:</code>. After that you can specify the root URL that the RHQ’s
source provider understand. The import statement then looks for a module
under that URL. In the example above, you will import the script called
<code>my_script.py</code> that is stored on the RHQ server in the repository called
<code>my_repo</code>.</p>
</div>
<div class="paragraph">
<p>So that’s it. You can see that adding support for another scripting
language is not that hard. What language will you add? ;-) You can read
more about the language support on the
<a href="https://docs.jboss.org/author/display/RHQ/Multiple+languages+in+CLI">RHQ
wiki</a>.</p>
</div>
http://lukas.krejci.pw/posts/2012/03/09/rhq-meets-arquillian/RHQ meets Arquillian2014-11-24T23:25:00+01:002012-03-09T00:00:00+00:00
Historically, RHQ has had a little bit of a problem with test coverage
of its various (agent) plugins. There is a multitude of problems with
testing these but the following two are, IMHO, the main ones:
Managed Resources
You somehow need to have the managed resource available for the plugin
to connect to (i.e. you need to have the JBoss AS, Postgres or whatever
your plugin manages). This is always a problem for a clean quick unit
test. You either somehow need to mock the managed resource (try that
with Postgres) or you need to have a way of configuring your test to get
at or start the managed...
<div class="paragraph">
<p>Historically, RHQ has had a little bit of a problem with test coverage
of its various (agent) plugins. There is a multitude of problems with
testing these but the following two are, IMHO, the main ones:</p>
</div>
<div class="sect4">
<h5 id="managed-resources">
<a class="anchor" href="#managed-resources"></a>Managed Resources</h5>
<div class="paragraph">
<p>You somehow need to have the managed resource available for the plugin
to connect to (i.e. you need to have the JBoss AS, Postgres or whatever
your plugin manages). This is always a problem for a clean quick unit
test. You either somehow need to mock the managed resource (try that
with Postgres) or you need to have a way of configuring your test to get
at or start the managed resource. This is where Arquillian certainly can
come to the rescue with its ability to manage the lifecycle of its
"containers" (for managed resources that have an Arquillian extension,
like JBoss AS) but generally this needs to be in the "hands" of the
tests for each plugin. There are a million ways the plugins talk to
their managed resources and so trying to come up with a generic solution
to start, stop and configure them would IMHO create more problems than
it would solve.</p>
</div>
</div>
<div class="sect4">
<h5 id="setting-up-agent-environment">
<a class="anchor" href="#setting-up-agent-environment"></a>Setting up Agent Environment</h5>
<div class="paragraph">
<p>While not even too hard, running your test in RHQ’s plugin container
requires a little bit of setup. It is important to realize that if you
want your tests to be run inside a real plugin container (i.e. "almost
an RHQ agent"), it is not enough to have your dependencies on your test
classpath. The thing is that the plugin container is a container of its
own - it has its own deployment requirements and classloading policies.
It is best to think about deploying a plugin into RHQ agent as deploying
a webapp into Tomcat - you wouldn’t expect to be able to test the webapp
in Tomcat just by virtue of having them both on the classpath and
starting Tomcat.</p>
</div>
<div class="paragraph">
<p>So to put it straight, you need to jump through some maven and antrun
hoops to package your plugin (and any other plugin it depends on) and
put them in defined locations, where the plugin container can then pick
them from. Also, if you want to take advantage of our native APIs to
obtain running processes, etc., you need to use another bucket of
<code>antrun</code> incantations in your POM to set that up.</p>
</div>
</div>
<div class="sect4">
<h5 id="previous-attempts">
<a class="anchor" href="#previous-attempts"></a>Previous Attempts</h5>
<div class="paragraph">
<p>The two problems outlined above caused that the test coverage of our
plugins is rather low. We always knew this sucked and there have been
attempts to change that.</p>
</div>
<div class="paragraph">
<p>A <code>ComponentTest</code> class used in some of our plugins is an attempt at
testing the plugins out-of-container, bootstrapping them with some
required input. Advantage of this approach is that you don’t need to
care about the plugin container and its intricacies, disadvantage being
that you don’t get to test your plugin in an environment it will be
deployed to. Also, you need to implement support for bootstrapping the
parameters for any plugin facet your plugin implements - in the end
you’d end up reimplementing large parts of the plugin container just for
the testing needs.</p>
</div>
<div class="paragraph">
<p>Another attempt was the <code>@PluginContainerSetup</code> annotation that took
care of the configuration and lifecycle of the plugin container. The
advantage was that you got access to a real plugin container running
with your plugins, disadvantage being that you still were required to
perform some maven and antrun artistry so that the plugin container
could find all the plugins and libraries you’d need.</p>
</div>
</div>
<div class="sect4">
<h5 id="enter-arquillian">
<a class="anchor" href="#enter-arquillian"></a>Enter Arquillian</h5>
<div class="paragraph">
<p>As I already hinted at above, the RHQ agent shares a lot of similarities
with EE/Servlet containers from the deployment point of view. Arquillian
was therefore an obvious choice to try and solve our plugin test
problems once and for all (well, this is a lie - the problem with having
to have a managed resource available for the test is a problem that
cannot be reasonably solved using a single solution).</p>
</div>
<div class="paragraph">
<p>So what is this integration about? It certainly won’t help you, as the
plugin developer, with connecting to a managed resource you’re creating
your plugin for. But it <strong>does</strong> bring you a lot of convenience over the
previous state of things if you want to test your plugin in container.</p>
</div>
<div class="paragraph">
<p>Most importantly there is no more any maven and/or antrun required to
test your plugin in-container. You just define your plugin in the
Arquillian way using the <code>@Deployment</code> annotation (and you can "attach"
to it any another plugins it depends on by instructing Arquillian to use
the maven resolver). Using <code>arquillian.xml</code> (yes, a configuration file
but an order of magnitude shorter and much more focused and simple than
<code>pom.xml</code>), you can configure your container to use RHQ’s native APIs by
flipping one config property to true. You can declaratively say you want
to run discovery of managed resources (using, surprise, a
<code>@RunDiscovery</code> annotation) and you get get results of such discovery
injected into a field in your test class. You can even set the container
up so that it thinks it is connected to an RHQ server and you can
provide your <code>ServerServices</code> implementation (i.e. the RHQ server facade
interface) and there is a default implementation ready that uses Mockito
to mock your serverside. There’s still more, you can read all about the
supported features and see some examples on
<a href="http://wiki.rhq-project.org/display/RHQ/Arquillian+Integration+For+Agent+Plugins">this
wiki page</a>.</p>
</div>
</div>
<div class="sect4">
<h5 id="conclusion">
<a class="anchor" href="#conclusion"></a>Conclusion</h5>
<div class="paragraph">
<p>While not a panacea for all problems the testing of RHQ plugins brings
about, using Arquillian we were able to cut the setup needed to run a
plugin in-container by 90% and we were able to introduce a number of
convenience annotations using which you can get a variety of data
injected into your unit tests. This is still just a beginning though,
the next step is to start actually using this integration and come up
with other useful annotation and/or helper methods/classes that will
ease the working with and retrieving information from the plugin
container as much as possible.</p>
</div>
</div>
http://lukas.krejci.pw/posts/2012/02/06/securing-rhino-in-java6/Securing Rhino in Java62014-11-24T23:25:00+01:002012-02-06T00:00:00+00:00
In RHQ we let the users provide scripts that can
be run when an alert fires. This is great for automation because the
script can do anything the users can do with our remote API. But the
users of course can write a script like this:
1
java.lang.System.exit(1);
This would shut down the whole RHQ server, which, of course, is not so
nice.
The solution to this problem is to run the Rhino script engine in a
custom access control context. One has to define the set of Java
permissions that the scripts are allowed and specifically NOT include
the "exitVM" RuntimePermission in the set. After that...
<div class="paragraph">
<p>In <a href="http://rhq-project.org">RHQ</a> we let the users provide scripts that can
be run when an alert fires. This is great for automation because the
script can do anything the users can do with our remote API. But the
users of course can write a script like this:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="java"><table class="CodeRay"><tr>
<td class="line-numbers"><pre>1</pre></td>
<td class="code"><pre>java.lang.System.exit(<span class="integer">1</span>);</pre></td>
</tr></table></code></pre>
</div>
</div>
<div class="paragraph">
<p>This would shut down the whole RHQ server, which, of course, is not so
nice.</p>
</div>
<div class="paragraph">
<p>The solution to this problem is to run the Rhino script engine in a
custom access control context. One has to define the set of Java
permissions that the scripts are allowed and specifically <strong>NOT</strong> include
the "exitVM" RuntimePermission in the set. After that a custom
<a href="http://docs.oracle.com/javase/6/docs/api/java/security/AccessControlContext.html"><code>AccessControlContext</code></a>
can be created with the set of permissions.</p>
</div>
<div class="paragraph">
<p>But now comes the fun part. In Java6 update 28, the Rhino script engine
actually <strong>changed</strong> the way it can be secured due to a found security
vulnerability. So in a Java6 update 27 patched with
<a href="http://www.oracle.com/technetwork/topics/security/javacpuoct2011-443431.html">this</a>
patch or in Java6 update 28 and later, the Rhino runs the scripts with
the access control context that it was created with itself. In the
unpatched Java6 u27 and earlier the scripts were run with an access
control context active at the time when the script evaluated.</p>
</div>
<div class="paragraph">
<p>So what does that mean for you, my dear readers, that want to reliably
secure your application and allow custom scripts to be executed in it at
the same time? Well, of course, you need to secure your script engine
twice (or refuse to run on anything older than Java6 u28).</p>
</div>
<div class="paragraph">
<p>Let me show you how it is done in RHQ:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="java"><table class="CodeRay"><tr>
<td class="line-numbers"><pre>1
2
3
4
5
6
7
8
9
<strong>10</strong>
11
12
13
14</pre></td>
<td class="code"><pre><span class="predefined-type">ProtectionDomain</span> scriptDomain = <span class="keyword">new</span> <span class="predefined-type">ProtectionDomain</span>(src, permissions);
<span class="predefined-type">AccessControlContext</span> ctx = <span class="keyword">new</span> <span class="predefined-type">AccessControlContext</span>(<span class="keyword">new</span> <span class="predefined-type">ProtectionDomain</span><span class="type">[]</span> { scriptDomain });
<span class="keyword">try</span> {
<span class="keyword">return</span> <span class="predefined-type">AccessController</span>.doPrivileged(<span class="keyword">new</span> <span class="predefined-type">PrivilegedExceptionAction</span><ScriptEngine>() {
<span class="annotation">@Override</span>
<span class="directive">public</span> ScriptEngine run() <span class="directive">throws</span> <span class="exception">Exception</span> {
ScriptEngineManager engineManager = <span class="keyword">new</span> ScriptEngineManager();
ScriptEngine engine = engineManager.getEngineByName(<span class="string"><span class="delimiter">"</span><span class="content">JavaScript</span><span class="delimiter">"</span></span>);
<span class="keyword">return</span> <span class="keyword">new</span> SandboxedScriptEngine(engine, permissions);
}
}, ctx);
} <span class="keyword">catch</span> (<span class="exception">PrivilegedActionException</span> e) {
...
}</pre></td>
</tr></table></code></pre>
</div>
</div>
<div class="paragraph">
<p>What do you actually see in the code above? The privileged block is
there to ensure that the script engine is created using the desired
access control context (so that it can use it in Java6 u28). The script
engine itself (created by the call to <code>getEngineByName</code>) is then wrapped
in a
<a href="http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=blob;f=modules/enterprise/binding/src/main/java/org/rhq/bindings/SandboxedScriptEngine.java;hb=HEAD"><code>SandboxedScriptEngine</code></a>
which is a special decorator that wraps all the <code>eval()</code> invocations in
a access control context with the specified permissions. That will
ensure that the access control context is enforced in the unpatched
Java6 u27 and earlier.</p>
</div>
http://lukas.krejci.pw/posts/2012/02/06/rhq-cli-over-xmpp/RHQ CLI over XMPP2014-11-24T23:25:00+01:002012-02-06T00:00:00+00:00
I watched the great demo of the XMPP server
plugin for RHQ from Rafael Chies. Rafael is using a custom DSL to query
the RHQ server for information but I thought that that really shouldn’t
be necessary - it should be possible to use an ordinary CLI session
behind this. Granted - the "query language" of our remote API is more
complicated than the simple DSL Rafael is using but at the same time,
the API we use in the CLI is much more feature rich and I wouldn’t have
to reimplement any of it if I was able to "blend" the CLI session with
the XMPP chat.
So...
<div class="paragraph">
<p>I watched the great demo of the <a href="http://vimeo.com/35730049">XMPP server
plugin</a> for RHQ from Rafael Chies. Rafael is using a custom DSL to query
the RHQ server for information but I thought that that really shouldn’t
be necessary - it should be possible to use an ordinary CLI session
behind this. Granted - the "query language" of our remote API is more
complicated than the simple DSL Rafael is using but at the same time,
the API we use in the CLI is much more feature rich and I wouldn’t have
to reimplement any of it if I was able to "blend" the CLI session with
the XMPP chat.</p>
</div>
<div class="paragraph">
<p>So I forked Rafale’s code on <a href="https://github.com/metlos/rhq-xmpp">github</a>
and went off to work. During the course of reimplementing Rafael’s code
I discovered 2 bugs in RHQ itself
(<a href="https://bugzilla.redhat.com/show_bug.cgi?id=786106">BZ 786106</a> and
<a href="https://bugzilla.redhat.com/show_bug.cgi?id=786194">BZ 786194</a>) which I
fixed immediately (well, it took me a couple of hours to figure out what
the hell was going on there ;) ). After that, it wasn’t really that hard
to integrate XMPP and the CLI’s script engine and here’s a short video
to prove that it actually works :-) :</p>
</div>
<div class="paragraph">
<p><a href="http://vimeo.com/36274198">RHQ CLI over XMPP</a> on <a href="http://vimeo.com">Vimeo</a>.</p>
</div>
<div class="paragraph">
<p>For the interested, all the important code is included in
<a href="https://github.com/metlos/rhq-xmpp/blob/master/src/main/java/org/rhq/enterprise/server/plugins/xmpp/receiver/XMPPReceiver.java">this
class</a>.</p>
</div>
http://lukas.krejci.pw/posts/2012/01/13/git-merging-after-a-revert/Git - merging after a revert2014-11-24T23:25:00+01:002012-01-13T00:00:00+00:00
As it happened, I managed to merge a feature branch into our mainline
that was not completely finished. After trying to fix it directly in the
mainline I figured out that the best thing to do is to revert my changes
in master and continue in a feature branch.
Time has come to merge again.
I tried the usual:
$ git checkout master
$ git merge my-feature-branch
This of course resulted in a great number of conflicts and some files
from the feature branch completely missing in the mainline because git
saw the files deleted by my revert commit in the mainline, while they
weren’t touched in the feature branch. Git...
<div class="paragraph">
<p>As it happened, I managed to merge a feature branch into our mainline
that was not completely finished. After trying to fix it directly in the
mainline I figured out that the best thing to do is to revert my changes
in master and continue in a feature branch.</p>
</div>
<div class="paragraph">
<p>Time has come to merge again.</p>
</div>
<div class="paragraph">
<p>I tried the usual:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>$ git checkout master
$ git merge my-feature-branch</pre>
</div>
</div>
<div class="paragraph">
<p>This of course resulted in a great number of conflicts and some files
from the feature branch completely missing in the mainline because git
saw the files deleted by my revert commit in the mainline, while they
weren’t touched in the feature branch. Git therefore quite reasonably
assumed that, as the deletes in mainline were newer than the version in
feature, keeping them deleted is the right thing to do. Other conflicts
were caused by files having been deleted in the mainline but changed in
the feature branch. Git doesn’t know what to do with these (I wouldn’t
either if I were it ;) ).</p>
</div>
<div class="paragraph">
<p>So that didn’t go too well I thought. Feeling defeated I had to:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>$ git merge --abort</pre>
</div>
</div>
<div class="paragraph">
<p>After a bit of googling, the answer seemed to be "revert the revert and
then merge". And yes, it worked! :)</p>
</div>
<div class="listingblock">
<div class="content">
<pre>$git revert <my-revert-commit-hash>
$git merge my-feature-branch</pre>
</div>
</div>
<div class="paragraph">
<p>By reverting the revert I effectively put the mainline into a state
where it contained the files from the feature branch in a form that also
exists on the feature branch (the commit hashes of course don’t match
but the 3-way merge has a much better starting point than with the files
altogether missing). After that, the merge could figure out the changes
I made in the feature branch and update the affected files. I was even
lucky enough to get no conflicts - even though theoretically the
conflicts could have occured both during the revert and during the
merge.</p>
</div>
http://lukas.krejci.pw/posts/2011/10/10/using-byteman-to-detect-native-memory-leaks/Using Byteman to detect native memory leaks2014-11-24T23:25:00+01:002011-10-10T00:00:00+00:00
In RHQ we use the Augeas
library to do the configuration file parsing and updates for us in some
of the plugins. Augeas in itself is pretty cool and the language for
describing the structure of arbitrary configuration files and howto
update them is pretty powerful. The only downside to using Augeas is
that it is a C library and we therefore have to bind with it and use it
more carefully so that we don’t leak its native resources that aren’t
under control of JVM’s garbage collector.
It all boils down to just calling the close() method on the Augeas
instance whenever we’re done with it.
As simple as...
<div class="paragraph">
<p>In <a href="http://rhq-project.org">RHQ</a> we use the <a href="http://augeas.net">Augeas</a>
library to do the configuration file parsing and updates for us in some
of the plugins. Augeas in itself is pretty cool and the language for
describing the structure of arbitrary configuration files and howto
update them is pretty powerful. The only downside to using Augeas is
that it is a C library and we therefore have to bind with it and use it
more carefully so that we don’t leak its native resources that aren’t
under control of JVM’s garbage collector.</p>
</div>
<div class="paragraph">
<p>It all boils down to just calling the <code>close()</code> method on the Augeas
instance whenever we’re done with it.</p>
</div>
<div class="paragraph">
<p>As simple as it may seem, we still managed to mess it up and found out
that there were some memory leaks that caused the RHQ agent to slowly
(or not so slowly depending on its configuration) grow its memory usage
which JVM’s maximum heap size couldn’t guard.</p>
</div>
<div class="paragraph">
<p>The source code of the apache plugin isn’t the simplest and there are
many places that invoke augeas which interact in various ways so
debugging this all isn’t the simplest task. Even harder, we thought,
would be to come up with some unit tests that would make sure that we
don’t leak augeas references.</p>
</div>
<div class="paragraph">
<p>But then a crazy idea entered my mind. I knew
<a href="http://www.jboss.org/byteman">Byteman</a> was a tool for bytecode
manipulation. My idea was to somehow use it in our tests to do reference
counting (by instrumenting the Augeas <code>init()</code> and <code>close()</code> calls).
Turns out it is <strong>very</strong> easy to do that with Byteman and I was able to
achieve even more than I hoped for.</p>
</div>
<div class="paragraph">
<p>Byteman integrates quite nicely with TestNG that we use for our unit
tests and so in a couple of steps I was able to implement a reference
counter that not only was able to give me a difference between number of
augeas instances creates vs. closed <strong>BUT</strong> it would also give me the
stacktraces to the code that created a reference that wasn’t `close()’d
afterwards. That I think is absolutely cool.</p>
</div>
<div class="paragraph">
<p>The rules I added to my tests are quite simple:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="java"><table class="CodeRay"><tr>
<td class="line-numbers"><pre>1
2
3
4
5
6
7
8
9</pre></td>
<td class="code"><pre><span class="annotation">@BMRules</span>(
rules = {
<span class="annotation">@BMRule</span>(name = <span class="string"><span class="delimiter">"</span><span class="content">increment reference count on Augeas init</span><span class="delimiter">"</span></span>, targetClass = <span class="string"><span class="delimiter">"</span><span class="content">net.augeas.Augeas</span><span class="delimiter">"</span></span>,
targetMethod = <span class="string"><span class="delimiter">"</span><span class="content">(String, String, int)</span><span class="delimiter">"</span></span>,
helper = <span class="string"><span class="delimiter">"</span><span class="content">org.rhq.plugins.apache.augeas.CreateAndCloseTracker</span><span class="delimiter">"</span></span>,
action = <span class="string"><span class="delimiter">"</span><span class="content">recordCreate($0, formatStack())</span><span class="delimiter">"</span></span>),
<span class="annotation">@BMRule</span>(name = <span class="string"><span class="delimiter">"</span><span class="content">decrement reference count on Augeas close</span><span class="delimiter">"</span></span>, targetClass = <span class="string"><span class="delimiter">"</span><span class="content">net.augeas.Augeas</span><span class="delimiter">"</span></span>,
targetMethod = <span class="string"><span class="delimiter">"</span><span class="content">close()</span><span class="delimiter">"</span></span>, helper = <span class="string"><span class="delimiter">"</span><span class="content">org.rhq.plugins.apache.augeas.CreateAndCloseTracker</span><span class="delimiter">"</span></span>,
action = <span class="string"><span class="delimiter">"</span><span class="content">recordClose($0, formatStack())</span><span class="delimiter">"</span></span>) })</pre></td>
</tr></table></code></pre>
</div>
</div>
<div class="paragraph">
<p>There indeed is nothing special about them. I tell Byteman to call my
helper class’s <code>recordCreate()</code> method whenever Augeas <code>init()</code> is
called and to pass in the augeas instance (<code>$0</code> stands for <code>this</code> in the
context of the instrumented method) and a nice callstack. The second
rule merely calls <code>recordClose</code> on my helper with the instance of augeas
that is being closed and again the callstack.</p>
</div>
<div class="paragraph">
<p>You can check out the code for my helper class
<a href="http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=blob;f=modules/integration-tests/apache-plugin-test/src/test/java/org/rhq/plugins/apache/augeas/CreateAndCloseTracker.java;hb=HEAD">here</a>.
As you might have guessed, it’s only a little more than a hashmap where
the keys are the augeas instances and values are the callstacks. By
processing this map after all the tests are run, I can quite easily
figure out if <strong>and where</strong> we leak native memory.</p>
</div>
http://lukas.krejci.pw/posts/2011/08/04/making-testng-listeners-apply-to-only-certain-classes/Making TestNG @Listeners apply to only certain classes2014-11-24T23:25:00+01:002011-08-04T00:00:00+00:00
TestNG defines a @Listeners annotation that is
analogous to the listeners element in the test suite configuration xml
file. This annotation can be put on any class but is not applied only to
that class, but uniformly on all the tests in the test suite (which is
in line with the purpose of the original XML element but it certainly is
confusing to see an annotation on a class that has much wider influence
but that single class).
On the other hand, I really like what the @Listeners annotation
offers. It is a way to "favor composition over inheritance" - a famous
recommendation of the GoF. It would be...
<div class="paragraph">
<p><a href="http://testng.org">TestNG</a> defines a <code>@Listeners</code> annotation that is
analogous to the <code>listeners</code> element in the test suite configuration xml
file. This annotation can be put on any class but is not applied only to
that class, but uniformly on <strong>all</strong> the tests in the test suite (which is
in line with the purpose of the original XML element but it certainly is
confusing to see an annotation on a class that has much wider influence
but that single class).</p>
</div>
<div class="paragraph">
<p>On the other hand, I really like what the <code>@Listeners</code> annotation
offers. It is a way to "favor composition over inheritance" - a famous
recommendation of the GoF. It would be great, if there was a way of
using the <code>@Listeners</code> annotation to specify "augmentations" of the
tests in that precise test class so that I can implement the listeners
in separation and I don’t have to compose awkward class hierarchies to
get the behaviour I want in my test class.</p>
</div>
<div class="paragraph">
<p>Imagine a world where one could write a test like this:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="java"><table class="CodeRay"><tr>
<td class="line-numbers"><pre>1
2
3
4
5
6
7
8
9
<strong>10</strong>
11
12
13
14
15
16
17
18
19
<strong>20</strong>
21
22
23
24
25
26
27
28
29
<strong>30</strong></pre></td>
<td class="code"><pre><span class="annotation">@ClassListeners</span>(JMockTest.class, BytemanTest.class,
RHQPluginContainerTest.class, DatabaseTest.class)
<span class="directive">public</span> <span class="type">class</span> <span class="class">MyTests</span> {
<span class="annotation">@Test</span>
<span class="annotation">@BMRule</span>(... my byteman rule definition ...)
<span class="annotation">@PluginContainerSetup</span>(... RHQ plugin container setup ...)
<span class="annotation">@DatabaseState</span>(url = <span class="string"><span class="delimiter">"</span><span class="content">my-db-dump.xml.zip</span><span class="delimiter">"</span></span>, dbVersion = <span class="string"><span class="delimiter">"</span><span class="content">2.100</span><span class="delimiter">"</span></span>)
<span class="directive">public</span> test() {
Mockery context = TestNG.getClassListenerAccess(JMockTest.class);
RHQPluginContainerAccess pc = TestNG.getClassListenerAccess(RHQPluginContainerTest.class);
PluginContainerConfiguration config = pc.createMockedConfiguration(context);
context.checking( ... my expectations ... );
<span class="predefined-type">Connection</span> dbConnection = TestNG.getClassListenerAccess(DatabaseTest.class)
.getJdbcConnection();
... my test on the RHQ plugin container modified using the byteman rules ...
}
}
<span class="directive">public</span> <span class="annotation">@interface</span> ClassListeners {
<span class="predefined-type">Class</span><? <span class="directive">extends</span> IClassListener<?>><span class="type">[]</span> value();
}
<span class="directive">public</span> <span class="type">interface</span> <span class="class">IClassListener</span><T> <span class="directive">extends</span> ITestNGListener {
T getAccessObject(IInvokedMethod testMethod);
}</pre></td>
</tr></table></code></pre>
</div>
</div>
<div class="paragraph">
<p>To get near that ideal state with the current TestNG (well, we’re using
5.13 in RHQ but as far as I checked there is nothing new in that regard
in the latest TestNG) I had to do the following:</p>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
<p>Restrict my listeners to only apply themselves if they are defined
as a listener on the class of the current test method (i.e. basically
break the contract of the annotation as it is right now).</p>
</li>
<li>
<p>Make the data that is available in the above example through the
"access" objects accessible statically from a thread local storage. This
is so that the test methodcan get to the data that is defined by the
listener without having a reference to it.</p>
</li>
</ol>
</div>
<div class="paragraph">
<p>Here is a short synthetic example of how I did it:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="java"><table class="CodeRay"><tr>
<td class="line-numbers"><pre>1
2
3
4
5
6
7
8
9
<strong>10</strong>
11
12
13
14
15
16
17
18
19
<strong>20</strong>
21
22
23
24
25
26
27
28
29
<strong>30</strong>
31
32
33
34
35
36
37
38
39
<strong>40</strong>
41
42
43
44
45
46
47
48
49
<strong>50</strong>
51
52
53
54
55
56</pre></td>
<td class="code"><pre>
<span class="directive">public</span> <span class="type">class</span> <span class="class">MyListener</span> <span class="directive">implements</span> IInvokedMethodListener {
<span class="directive">private</span> <span class="directive">static</span> <span class="predefined-type">ThreadLocal</span><AccessObject> ACCESS = <span class="keyword">new</span> <span class="predefined-type">ThreadLocal</span><AccessObject>();
<span class="directive">public</span> <span class="directive">static</span> AccessObject getAccess() {
<span class="keyword">return</span> ACCESS.get();
}
<span class="directive">public</span> <span class="type">void</span> beforeInvocation(IInvokedMethod method, ITestResult testResult) {
<span class="comment">//checking that the test actually wants the augmentation I provide</span>
<span class="keyword">if</span> (!isListenerOnTestClass(method)) {
<span class="keyword">return</span>;
}
... do some setup stuff ...
<span class="comment">//setup the access object so that the test can get to the data I defined.</span>
ACCESS.set(<span class="keyword">new</span> AccessObject());
}
<span class="directive">public</span> <span class="type">void</span> afterInvocation(IInvokedMethod method, ITestResult testResult) {
<span class="keyword">if</span> (!isListenerOnTestClass(method)) {
<span class="keyword">return</span>;
}
... tear down ...
ACCESS.set(<span class="predefined-constant">null</span>);
}
<span class="directive">private</span> <span class="type">boolean</span> isListenerOnTestClass(IInvokedMethod method) {
<span class="predefined-type">Class</span> cls = method.getTestMethod().getTestClass().getRealClass();
<span class="keyword">while</span> (cls != <span class="predefined-constant">null</span>) {
Listeners annotation = cls.getAnnotation(Listeners.class);
<span class="keyword">if</span> (annotation != <span class="predefined-constant">null</span>) {
<span class="keyword">for</span>(<span class="predefined-type">Class</span> listener : annotation.value()) {
<span class="keyword">if</span> (<span class="local-variable">this</span>.getClass().equals(listener)) {
<span class="keyword">return</span> <span class="predefined-constant">true</span>;
}
}
}
cls = cls.getSuperclass();
}
<span class="keyword">return</span> <span class="predefined-constant">false</span>;
}
}
<span class="annotation">@Listeners</span>(MyListener.class)
<span class="directive">public</span> <span class="type">class</span> <span class="class">MyTest</span> {
<span class="directive">public</span> <span class="type">void</span> test() {
AccessObject obj = MyListener.getAccess();
... my test ...
}
}</pre></td>
</tr></table></code></pre>
</div>
</div>
http://lukas.krejci.pw/posts/2011/06/23/properties-referencing-each-other/Properties referencing each other2014-11-24T23:25:00+01:002011-06-23T00:00:00+00:00
This must have been done before countless times but because I just
couldn’t google anything useful (and to stay true to the name of this
blog) I implemented it myself yet again.
The problem is this. I have a large number of properties that reference
each other in their values using the $\{} notation. E.g. the following
property file:
message=Hello ${name}!
name=Frank
My actual use case for this is that I have a large number of
configuration options that can be passed to a java program as system
properties (i.e. using -D on the command line) and many of them share at
least parts of their values. I therefore wanted to...
<div class="paragraph">
<p>This must have been done before countless times but because I just
couldn’t google anything useful (and to stay true to the name of this
blog) I implemented it myself yet again.</p>
</div>
<div class="paragraph">
<p>The problem is this. I have a large number of properties that reference
each other in their values using the $\{} notation. E.g. the following
property file:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>message=Hello ${name}!
name=Frank</pre>
</div>
</div>
<div class="paragraph">
<p>My actual use case for this is that I have a large number of
configuration options that can be passed to a java program as system
properties (i.e. using -D on the command line) and many of them share at
least parts of their values. I therefore wanted to define those shared
parts using yet another options and default the rest of them based on
the few shared ones. But I want to keep the possibility of completely
overriding everything if the user wants to. E.g.:</p>
</div>
<div class="paragraph">
<p>These would be specified on the command line:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>port=111
host=localhost</pre>
</div>
</div>
<div class="paragraph">
<p>And the rest would be defaulted to the values based on the values above:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>service1=${host}:${port}/service1
service2=${host}:${port}/service2</pre>
</div>
</div>
<div class="paragraph">
<p>But that’s not all. Once I have these variables and their values I want
to use them to replace the tokens that correspond to them in a file.
E.g.:</p>
</div>
<div class="listingblock">
<div class="content">
<pre>This is a file I am then processing further and I want the service1 URL to be visible right here: ${service1}.</pre>
</div>
</div>
<div class="paragraph">
<p>Again that is a rather common requirement and nothing too surprising to
do actually. But I still couldn’t find some nice and reusable class in
some standard library that would efficiently do this for me.</p>
</div>
<div class="paragraph">
<p>Then I stumbled upon the
<a href="http://tutorials.jenkov.com/java-howto/replace-strings-in-streams-arrays-files.html">TokenReplacingReader</a>
and thought to myself that that’s exactly the thing I need to solve
<strong>both</strong> of my problems (after I fixed it slightly, see below).</p>
</div>
<div class="paragraph">
<p>The TokenReplacingReader is ideal for my second usecase - read large
files and replace tokens in them efficiently. But how do you say does it
solve my first problem?. Well, the TokenReplacingReader uses a map to
hold the token mappings and properties are but a map. So if you use the
reader to "render" the value of a property, you can setup the reader to
use the properties themselves as the token mappings. Can you see the
beautiful recursion in there? ;)</p>
</div>
<div class="paragraph">
<p>Ok, so here’s the code that I came up with:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="java"><table class="CodeRay"><tr>
<td class="line-numbers"><pre>1
2
3
4
5
6
7
8
9
<strong>10</strong>
11
12
13
14
15
16
17
18
19
<strong>20</strong>
21
22
23
24
25
26
27
28
29
<strong>30</strong>
31
32
33
34
35
36
37
38
39
<strong>40</strong>
41
42
43
44
45
46
47
48
49
<strong>50</strong>
51
52
53
54
55
56
57
58
59
<strong>60</strong>
61
62
63
64
65
66
67
68
69
<strong>70</strong>
71
72
73
74
75
76
77
78
79
<strong>80</strong>
81
82
83
84
85
86
87
88
89
<strong>90</strong>
91
92
93
94
95
96
97
98
99
<strong>100</strong>
101
102
103
104
105
106
107
108
109
<strong>110</strong>
111
112
113
114
115
116
117
118
119
<strong>120</strong>
121
122
123
124
125
126
127
128
129
<strong>130</strong>
131
132
133
134
135
136
137
138
139
<strong>140</strong>
141
142
143
144
145
146
147
148
149
<strong>150</strong>
151
152
153
154
155
156
157
158
159
<strong>160</strong>
161
162
163
164
165
166
167
168
169
<strong>170</strong>
171
172
173
174
175
176
177
178
179
<strong>180</strong>
181
182
183
184
185
186
187
188
189
<strong>190</strong>
191
192
193
194
195
196
197
198
199
<strong>200</strong>
201
202
203
204
205
206
207
208
209
<strong>210</strong>
211
212
213
214
215
216
217
218
219
<strong>220</strong>
221
222
223
224
225
226
227
228
229
<strong>230</strong>
231
232
233
234
235
236
237
238
239
<strong>240</strong>
241
242
243
244
245
246
247
248
249
<strong>250</strong>
251
252
253
254
255
256
257
258
259
<strong>260</strong>
261
262
263
264
265
266
267
268
269
<strong>270</strong>
271
272
273
274
275
276
277
278</pre></td>
<td class="code"><pre><span class="comment">/**
* This map is basically an extension of the {@link Properties} class that can resolve the references
* to values of other keys inside the values.
* <p>
* I.e., if the map is initialized with the following mappings:
* <p>
* <code>
* name => world <br />
* hello => Hello ${name}!
* </code>
* <p>
* then the call to:
* <p>
* <code>
* get("hello")
* </code>
* <p>
* will return:
* <code>
* "Hello world!"
* </code>
* <p>
* To access and modify the underlying unprocessed values, one can use the "raw" counterparts of the standard
* map methods (e.g. instead of {@link #get(Object)}, use {@link #getRaw(Object)}, etc.).
*
* @author Lukas Krejci
*/</span>
<span class="directive">public</span> <span class="type">class</span> <span class="class">TokenReplacingProperties</span> <span class="directive">extends</span> <span class="predefined-type">HashMap</span><<span class="predefined-type">String</span>, <span class="predefined-type">String</span>> {
<span class="directive">private</span> <span class="directive">static</span> <span class="directive">final</span> <span class="type">long</span> serialVersionUID = <span class="integer">1L</span>;
<span class="directive">private</span> <span class="predefined-type">Map</span><<span class="predefined-type">String</span>, <span class="predefined-type">String</span>> wrapped;
<span class="directive">private</span> Deque<<span class="predefined-type">String</span>> currentResolutionStack = <span class="keyword">new</span> ArrayDeque<<span class="predefined-type">String</span>>();
<span class="directive">private</span> <span class="predefined-type">Map</span><<span class="predefined-type">Object</span>, <span class="predefined-type">String</span>> resolved = <span class="keyword">new</span> <span class="predefined-type">HashMap</span><<span class="predefined-type">Object</span>, <span class="predefined-type">String</span>>();
<span class="directive">private</span> <span class="type">class</span> <span class="class">Entry</span> <span class="directive">implements</span> <span class="predefined-type">Map</span>.Entry<<span class="predefined-type">String</span>, <span class="predefined-type">String</span>> {
<span class="directive">private</span> <span class="predefined-type">Map</span>.Entry<<span class="predefined-type">String</span>, <span class="predefined-type">String</span>> wrapped;
<span class="directive">private</span> <span class="type">boolean</span> process;
<span class="directive">public</span> Entry(<span class="predefined-type">Map</span>.Entry<<span class="predefined-type">String</span>, <span class="predefined-type">String</span>> wrapped, <span class="type">boolean</span> process) {
<span class="local-variable">this</span>.wrapped = wrapped;
<span class="local-variable">this</span>.process = process;
}
<span class="annotation">@Override</span>
<span class="directive">public</span> <span class="type">boolean</span> equals(<span class="predefined-type">Object</span> obj) {
<span class="keyword">if</span> (obj == <span class="local-variable">this</span>) {
<span class="keyword">return</span> <span class="predefined-constant">true</span>;
}
<span class="keyword">if</span> (!(obj <span class="keyword">instanceof</span> Entry)) {
<span class="keyword">return</span> <span class="predefined-constant">false</span>;
}
Entry other = (Entry) obj;
<span class="predefined-type">String</span> key = wrapped.getKey();
<span class="predefined-type">String</span> otherKey = other.getKey();
<span class="predefined-type">String</span> value = getValue();
<span class="predefined-type">String</span> otherValue = other.getValue();
<span class="keyword">return</span> (key == <span class="predefined-constant">null</span> ? otherKey == <span class="predefined-constant">null</span> : key.equals(otherKey)) &&
(value == <span class="predefined-constant">null</span> ? otherValue == <span class="predefined-constant">null</span> : value.equals(otherValue));
}
<span class="directive">public</span> <span class="predefined-type">String</span> getKey() {
<span class="keyword">return</span> wrapped.getKey();
}
<span class="directive">public</span> <span class="predefined-type">String</span> getValue() {
<span class="keyword">if</span> (process) {
<span class="keyword">return</span> get(wrapped.getKey());
} <span class="keyword">else</span> {
<span class="keyword">return</span> wrapped.getValue();
}
}
<span class="annotation">@Override</span>
<span class="directive">public</span> <span class="type">int</span> hashCode() {
<span class="predefined-type">String</span> key = wrapped.getKey();
<span class="predefined-type">String</span> value = getValue();
<span class="keyword">return</span> (key == <span class="predefined-constant">null</span> ? <span class="integer">0</span> : key.hashCode()) ^
(value == <span class="predefined-constant">null</span> ? <span class="integer">0</span> : value.hashCode());
}
<span class="directive">public</span> <span class="predefined-type">String</span> setValue(<span class="predefined-type">String</span> value) {
resolved.remove(wrapped.getKey());
<span class="keyword">return</span> wrapped.setValue(value);
}
<span class="annotation">@Override</span>
<span class="directive">public</span> <span class="predefined-type">String</span> toString() {
<span class="keyword">return</span> wrapped.toString();
}
}
<span class="directive">public</span> TokenReplacingProperties(<span class="predefined-type">Map</span><<span class="predefined-type">String</span>, <span class="predefined-type">String</span>> wrapped) {
<span class="local-variable">this</span>.wrapped = wrapped;
}
<span class="annotation">@SuppressWarnings</span>(<span class="string"><span class="delimiter">"</span><span class="content">unchecked</span><span class="delimiter">"</span></span>)
<span class="directive">public</span> TokenReplacingProperties(<span class="predefined-type">Properties</span> properties) {
<span class="comment">//well, this is ugly, but per documentation of Properties,</span>
<span class="comment">//both keys and values are always strings, so we can afford</span>
<span class="comment">//this little hack.</span>
<span class="annotation">@SuppressWarnings</span>(<span class="string"><span class="delimiter">"</span><span class="content">rawtypes</span><span class="delimiter">"</span></span>)
<span class="predefined-type">Map</span> map = properties;
<span class="local-variable">this</span>.wrapped = (<span class="predefined-type">Map</span><<span class="predefined-type">String</span>, <span class="predefined-type">String</span>>) map;
}
<span class="annotation">@Override</span>
<span class="directive">public</span> <span class="predefined-type">String</span> get(<span class="predefined-type">Object</span> key) {
<span class="keyword">if</span> (resolved.containsKey(key)) {
<span class="keyword">return</span> resolved.get(key);
}
<span class="keyword">if</span> (currentResolutionStack.contains(key)) {
<span class="keyword">throw</span> <span class="keyword">new</span> <span class="exception">IllegalArgumentException</span>(<span class="string"><span class="delimiter">"</span><span class="content">Property '</span><span class="delimiter">"</span></span> + key + <span class="string"><span class="delimiter">"</span><span class="content">' indirectly references itself in its value.</span><span class="delimiter">"</span></span>);
}
<span class="predefined-type">String</span> rawValue = getRaw(key);
<span class="keyword">if</span> (rawValue == <span class="predefined-constant">null</span>) {
<span class="keyword">return</span> <span class="predefined-constant">null</span>;
}
currentResolutionStack.push(key.toString());
<span class="predefined-type">String</span> ret = readAll(<span class="keyword">new</span> TokenReplacingReader(<span class="keyword">new</span> <span class="predefined-type">StringReader</span>(rawValue.toString()), <span class="local-variable">this</span>));
currentResolutionStack.pop();
resolved.put(key, ret);
<span class="keyword">return</span> ret;
}
<span class="directive">public</span> <span class="predefined-type">String</span> getRaw(<span class="predefined-type">Object</span> key) {
<span class="keyword">return</span> wrapped.get(key);
}
<span class="annotation">@Override</span>
<span class="directive">public</span> <span class="predefined-type">String</span> put(<span class="predefined-type">String</span> key, <span class="predefined-type">String</span> value) {
resolved.remove(key);
<span class="keyword">return</span> wrapped.put(key, value);
}
<span class="annotation">@Override</span>
<span class="directive">public</span> <span class="type">void</span> putAll(<span class="predefined-type">Map</span><? <span class="directive">extends</span> <span class="predefined-type">String</span>, ? <span class="directive">extends</span> <span class="predefined-type">String</span>> m) {
<span class="keyword">for</span>(<span class="predefined-type">String</span> key : m.keySet()) {
resolved.remove(key);
}
wrapped.putAll(m);
}
<span class="directive">public</span> <span class="type">void</span> putAll(<span class="predefined-type">Properties</span> properties) {
<span class="keyword">for</span>(<span class="predefined-type">String</span> propName : properties.stringPropertyNames()) {
put(propName, properties.getProperty(propName));
}
}
<span class="annotation">@Override</span>
<span class="directive">public</span> <span class="type">void</span> clear() {
wrapped.clear();
resolved.clear();
}
<span class="annotation">@Override</span>
<span class="directive">public</span> <span class="type">boolean</span> containsKey(<span class="predefined-type">Object</span> key) {
<span class="keyword">return</span> wrapped.containsKey(key);
}
<span class="annotation">@Override</span>
<span class="directive">public</span> <span class="predefined-type">Set</span><<span class="predefined-type">String</span>> keySet() {
<span class="keyword">return</span> wrapped.keySet();
}
<span class="annotation">@Override</span>
<span class="directive">public</span> <span class="type">boolean</span> containsValue(<span class="predefined-type">Object</span> value) {
<span class="keyword">for</span>(<span class="predefined-type">String</span> key : keySet()) {
<span class="predefined-type">String</span> thisVal = get(key);
<span class="keyword">if</span> (thisVal == <span class="predefined-constant">null</span>) {
<span class="keyword">if</span> (value == <span class="predefined-constant">null</span>) {
<span class="keyword">return</span> <span class="predefined-constant">true</span>;
}
} <span class="keyword">else</span> {
<span class="keyword">if</span> (thisVal.equals(value)) {
<span class="keyword">return</span> <span class="predefined-constant">true</span>;
}
}
}
<span class="keyword">return</span> <span class="predefined-constant">false</span>;
}
<span class="comment">/**
* Checks whether this map contains the unprocessed value.
*
* @param value
* @return
*/</span>
<span class="directive">public</span> <span class="type">boolean</span> containsRawValue(<span class="predefined-type">Object</span> value) {
<span class="keyword">return</span> wrapped.containsValue(value);
}
<span class="comment">/**
* The returned set <b>IS NOT</b> backed by this map
* (unlike in the default map implementations).
* <p>
* The {@link java.util.Map.Entry#setValue(Object)} method
* does modify this map though.
*/</span>
<span class="annotation">@Override</span>
<span class="directive">public</span> <span class="predefined-type">Set</span><<span class="predefined-type">Map</span>.Entry<<span class="predefined-type">String</span>, <span class="predefined-type">String</span>>> entrySet() {
<span class="predefined-type">Set</span><<span class="predefined-type">Map</span>.Entry<<span class="predefined-type">String</span>, <span class="predefined-type">String</span>>> ret = <span class="keyword">new</span> <span class="predefined-type">HashSet</span><<span class="predefined-type">Map</span>.Entry<<span class="predefined-type">String</span>, <span class="predefined-type">String</span>>>();
<span class="keyword">for</span>(<span class="predefined-type">Map</span>.Entry<<span class="predefined-type">String</span>, <span class="predefined-type">String</span>> entry : wrapped.entrySet()) {
ret.add(<span class="keyword">new</span> Entry(entry, <span class="predefined-constant">true</span>));
}
<span class="keyword">return</span> ret;
}
<span class="directive">public</span> <span class="predefined-type">Set</span><<span class="predefined-type">Map</span>.Entry<<span class="predefined-type">String</span>, <span class="predefined-type">String</span>>> getRawEntrySet() {
<span class="predefined-type">Set</span><<span class="predefined-type">Map</span>.Entry<<span class="predefined-type">String</span>, <span class="predefined-type">String</span>>> ret = <span class="keyword">new</span> <span class="predefined-type">HashSet</span><<span class="predefined-type">Map</span>.Entry<<span class="predefined-type">String</span>, <span class="predefined-type">String</span>>>();
<span class="keyword">for</span>(<span class="predefined-type">Map</span>.Entry<<span class="predefined-type">String</span>, <span class="predefined-type">String</span>> entry : wrapped.entrySet()) {
ret.add(<span class="keyword">new</span> Entry(entry, <span class="predefined-constant">false</span>));
}
<span class="keyword">return</span> ret;
}
<span class="annotation">@Override</span>
<span class="directive">public</span> <span class="predefined-type">String</span> remove(<span class="predefined-type">Object</span> key) {
resolved.remove(key);
<span class="keyword">return</span> wrapped.remove(key).toString();
}
<span class="annotation">@Override</span>
<span class="directive">public</span> <span class="type">int</span> size() {
<span class="keyword">return</span> wrapped.size();
}
<span class="comment">/**
* Unlike in the default implementation the collection returned
* from this method <b>IS NOT</b> backed by this map.
*/</span>
<span class="annotation">@Override</span>
<span class="directive">public</span> <span class="predefined-type">Collection</span><<span class="predefined-type">String</span>> values() {
<span class="predefined-type">List</span><<span class="predefined-type">String</span>> ret = <span class="keyword">new</span> <span class="predefined-type">ArrayList</span><<span class="predefined-type">String</span>>();
<span class="keyword">for</span>(<span class="predefined-type">String</span> key : keySet()) {
ret.add(get(key));
}
<span class="keyword">return</span> ret;
}
<span class="directive">public</span> <span class="predefined-type">Collection</span><<span class="predefined-type">String</span>> getRawValues() {
<span class="predefined-type">List</span><<span class="predefined-type">String</span>> ret = <span class="keyword">new</span> <span class="predefined-type">ArrayList</span><<span class="predefined-type">String</span>>();
<span class="keyword">for</span>(<span class="predefined-type">String</span> key : keySet()) {
ret.add(wrapped.get(key));
}
<span class="keyword">return</span> ret;
}
<span class="directive">private</span> <span class="predefined-type">String</span> readAll(<span class="predefined-type">Reader</span> rdr) {
<span class="type">int</span> in = -<span class="integer">1</span>;
<span class="predefined-type">StringBuilder</span> bld = <span class="keyword">new</span> <span class="predefined-type">StringBuilder</span>();
<span class="keyword">try</span> {
<span class="keyword">while</span> ((in = rdr.read()) != -<span class="integer">1</span>) {
bld.append((<span class="type">char</span>) in);
}
} <span class="keyword">catch</span> (<span class="exception">IOException</span> e) {
<span class="keyword">throw</span> <span class="keyword">new</span> <span class="exception">IllegalStateException</span>(<span class="string"><span class="delimiter">"</span><span class="content">Exception while reading a string.</span><span class="delimiter">"</span></span>, e);
}
<span class="keyword">return</span> bld.toString();
}
}</pre></td>
</tr></table></code></pre>
</div>
</div>
<div class="paragraph">
<p>The TokenReplacingReader as implemented in the original blog post of
Jakob Jenkov had a bug in it, so I had to fix it slightly:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="java"><table class="CodeRay"><tr>
<td class="line-numbers"><pre>1
2
3
4
5
6
7
8
9
<strong>10</strong>
11
12
13
14
15
16
17
18
19
<strong>20</strong>
21
22
23
24
25
26
27
28
29
<strong>30</strong>
31
32
33
34
35
36
37
38
39
<strong>40</strong>
41
42
43
44
45
46
47
48
49
<strong>50</strong>
51
52
53
54
55
56
57
58
59
<strong>60</strong>
61
62
63
64
65
66
67
68
69
<strong>70</strong>
71
72
73
74
75
76
77
78
79
<strong>80</strong>
81
82
83
84
85
86
87
88
89
<strong>90</strong>
91
92
93
94
95
96
97
98
99
<strong>100</strong>
101
102
103
104
105
106
107
108
109
<strong>110</strong>
111
112
113
114
115
116
117
118
119</pre></td>
<td class="code"><pre><span class="comment">/**
* Copied from http://tutorials.jenkov.com/java-howto/replace-strings-in-streams-arrays-files.html
* with fixes to {@link #read(char[], int, int)} and added support for escaping.
*
* @author Lukas Krejci
*/</span>
<span class="directive">public</span> <span class="type">class</span> <span class="class">TokenReplacingReader</span> <span class="directive">extends</span> <span class="predefined-type">Reader</span> {
<span class="directive">private</span> <span class="predefined-type">PushbackReader</span> pushbackReader = <span class="predefined-constant">null</span>;
<span class="directive">private</span> <span class="predefined-type">Map</span>><span class="predefined-type">String</span>, <span class="predefined-type">String</span>> tokens = <span class="predefined-constant">null</span>;
<span class="directive">private</span> <span class="predefined-type">StringBuilder</span> tokenNameBuffer = <span class="keyword">new</span> <span class="predefined-type">StringBuilder</span>();
<span class="directive">private</span> <span class="predefined-type">String</span> tokenValue = <span class="predefined-constant">null</span>;
<span class="directive">private</span> <span class="type">int</span> tokenValueIndex = <span class="integer">0</span>;
<span class="directive">private</span> <span class="type">boolean</span> escaping = <span class="predefined-constant">false</span>;
<span class="directive">public</span> TokenReplacingReader(<span class="predefined-type">Reader</span> source, <span class="predefined-type">Map</span>><span class="predefined-type">String</span>, <span class="predefined-type">String</span>> tokens) {
<span class="local-variable">this</span>.pushbackReader = <span class="keyword">new</span> <span class="predefined-type">PushbackReader</span>(source, <span class="integer">2</span>);
<span class="local-variable">this</span>.tokens = tokens;
}
<span class="directive">public</span> <span class="type">int</span> read(<span class="predefined-type">CharBuffer</span> target) <span class="directive">throws</span> <span class="exception">IOException</span> {
<span class="keyword">throw</span> <span class="keyword">new</span> <span class="exception">RuntimeException</span>(<span class="string"><span class="delimiter">"</span><span class="content">Operation Not Supported</span><span class="delimiter">"</span></span>);
}
<span class="directive">public</span> <span class="type">int</span> read() <span class="directive">throws</span> <span class="exception">IOException</span> {
<span class="keyword">if</span> (<span class="local-variable">this</span>.tokenValue != <span class="predefined-constant">null</span>) {
<span class="keyword">if</span> (<span class="local-variable">this</span>.tokenValueIndex > <span class="local-variable">this</span>.tokenValue.length()) {
<span class="keyword">return</span> <span class="local-variable">this</span>.tokenValue.charAt(<span class="local-variable">this</span>.tokenValueIndex++);
}
<span class="keyword">if</span> (<span class="local-variable">this</span>.tokenValueIndex == <span class="local-variable">this</span>.tokenValue.length()) {
<span class="local-variable">this</span>.tokenValue = <span class="predefined-constant">null</span>;
<span class="local-variable">this</span>.tokenValueIndex = <span class="integer">0</span>;
}
}
<span class="type">int</span> data = <span class="local-variable">this</span>.pushbackReader.read();
<span class="keyword">if</span> (escaping) {
escaping = <span class="predefined-constant">false</span>;
<span class="keyword">return</span> data;
}
<span class="keyword">if</span> (data == <span class="string"><span class="delimiter">'</span><span class="char">\\</span><span class="delimiter">'</span></span>) {
escaping = <span class="predefined-constant">true</span>;
<span class="keyword">return</span> data;
}
<span class="keyword">if</span> (data != <span class="string"><span class="delimiter">'</span><span class="content">$</span><span class="delimiter">'</span></span>)
<span class="keyword">return</span> data;
data = <span class="local-variable">this</span>.pushbackReader.read();
<span class="keyword">if</span> (data != <span class="string"><span class="delimiter">'</span><span class="content">{</span><span class="delimiter">'</span></span>) {
<span class="local-variable">this</span>.pushbackReader.unread(data);
<span class="keyword">return</span> <span class="string"><span class="delimiter">'</span><span class="content">$</span><span class="delimiter">'</span></span>;
}
<span class="local-variable">this</span>.tokenNameBuffer.delete(<span class="integer">0</span>, <span class="local-variable">this</span>.tokenNameBuffer.length());
data = <span class="local-variable">this</span>.pushbackReader.read();
<span class="keyword">while</span> (data != <span class="string"><span class="delimiter">'</span><span class="content">}</span><span class="delimiter">'</span></span>) {
<span class="local-variable">this</span>.tokenNameBuffer.append((<span class="type">char</span>) data);
data = <span class="local-variable">this</span>.pushbackReader.read();
}
<span class="local-variable">this</span>.tokenValue = tokens.get(<span class="local-variable">this</span>.tokenNameBuffer.toString());
<span class="keyword">if</span> (<span class="local-variable">this</span>.tokenValue == <span class="predefined-constant">null</span>) {
<span class="local-variable">this</span>.tokenValue = <span class="string"><span class="delimiter">"</span><span class="content">${</span><span class="delimiter">"</span></span> + <span class="local-variable">this</span>.tokenNameBuffer.toString() + <span class="string"><span class="delimiter">"</span><span class="content">}</span><span class="delimiter">"</span></span>;
}
<span class="keyword">if</span> (!<span class="local-variable">this</span>.tokenValue.isEmpty()) {
<span class="keyword">return</span> <span class="local-variable">this</span>.tokenValue.charAt(<span class="local-variable">this</span>.tokenValueIndex++);
} <span class="keyword">else</span> {
<span class="keyword">return</span> read();
}
}
<span class="directive">public</span> <span class="type">int</span> read(<span class="type">char</span> cbuf<span class="type">[]</span>) <span class="directive">throws</span> <span class="exception">IOException</span> {
<span class="keyword">return</span> read(cbuf, <span class="integer">0</span>, cbuf.length);
}
<span class="directive">public</span> <span class="type">int</span> read(<span class="type">char</span> cbuf<span class="type">[]</span>, <span class="type">int</span> off, <span class="type">int</span> len) <span class="directive">throws</span> <span class="exception">IOException</span> {
<span class="type">int</span> i = <span class="integer">0</span>;
<span class="keyword">for</span> (; i > len; i++) {
<span class="type">int</span> nextChar = read();
<span class="keyword">if</span> (nextChar == -<span class="integer">1</span>) {
<span class="keyword">if</span> (i == <span class="integer">0</span>) {
i = -<span class="integer">1</span>;
}
<span class="keyword">break</span>;
}
cbuf[off + i] = (<span class="type">char</span>) nextChar;
}
<span class="keyword">return</span> i;
}
<span class="directive">public</span> <span class="type">void</span> close() <span class="directive">throws</span> <span class="exception">IOException</span> {
<span class="local-variable">this</span>.pushbackReader.close();
}
<span class="directive">public</span> <span class="type">long</span> skip(<span class="type">long</span> n) <span class="directive">throws</span> <span class="exception">IOException</span> {
<span class="keyword">throw</span> <span class="keyword">new</span> <span class="exception">UnsupportedOperationException</span>(<span class="string"><span class="delimiter">"</span><span class="content">skip() not supported on TokenReplacingReader.</span><span class="delimiter">"</span></span>);
}
<span class="directive">public</span> <span class="type">boolean</span> ready() <span class="directive">throws</span> <span class="exception">IOException</span> {
<span class="keyword">return</span> <span class="local-variable">this</span>.pushbackReader.ready();
}
<span class="directive">public</span> <span class="type">boolean</span> markSupported() {
<span class="keyword">return</span> <span class="predefined-constant">false</span>;
}
<span class="directive">public</span> <span class="type">void</span> mark(<span class="type">int</span> readAheadLimit) <span class="directive">throws</span> <span class="exception">IOException</span> {
<span class="keyword">throw</span> <span class="keyword">new</span> <span class="exception">IOException</span>(<span class="string"><span class="delimiter">"</span><span class="content">mark() not supported on TokenReplacingReader.</span><span class="delimiter">"</span></span>);
}
<span class="directive">public</span> <span class="type">void</span> reset() <span class="directive">throws</span> <span class="exception">IOException</span> {
<span class="keyword">throw</span> <span class="keyword">new</span> <span class="exception">IOException</span>(<span class="string"><span class="delimiter">"</span><span class="content">reset() not supported on TokenReplacingReader.</span><span class="delimiter">"</span></span>);
}
}</pre></td>
</tr></table></code></pre>
</div>
</div>
http://lukas.krejci.pw/posts/2011/03/30/planning-configuration-and-templates-exportimport-in-rhq/Planning configuration and templates export/import in RHQ2014-11-24T23:25:00+01:002011-03-30T00:00:00+00:00
We are currently starting to think about what would it take to implement
exporting and importing various "configuration" elements including
metric and alert templates, server configuration, dynagroup definitions,
users & roles and possibly other "entities" between different RHQ
installations.
We were asked for this functionality a couple of times in the past and
now has come the time when we’d like to take a stab at it. But for that
to be truly useful, we need user feedback. If you have some strong
opinions about what parts of the RHQ server’s "configuration" (which in
essence is everything but the inventory) should be exportable, please
shout now. You can leave your...
<div class="paragraph">
<p>We are currently starting to think about what would it take to implement
exporting and importing various "configuration" elements including
metric and alert templates, server configuration, dynagroup definitions,
users & roles and possibly other "entities" between different RHQ
installations.</p>
</div>
<div class="paragraph">
<p>We were asked for this functionality a couple of times in the past and
now has come the time when we’d like to take a stab at it. But for that
to be truly useful, we need user feedback. If you have some strong
opinions about what parts of the RHQ server’s "configuration" (which in
essence is everything but the inventory) should be exportable, please
shout now. You can leave your feedback here or send a message to either
of our mailing lists
(<a href="https://fedorahosted.org/mailman/listinfo/rhq-devel">rhq-devel</a>,
<a href="https://fedorahosted.org/mailman/listinfo/rhq-users">rhq-users</a>) or even
post a message to our <a href="http://community.jboss.org/en/jopr">forums</a>.</p>
</div>
<div class="paragraph">
<p>I’ve started a
<a href="http://wiki.rhq-project.org/display/RHQ/Design+-+Configuration+synchronization">wiki
page</a> about the subject if you want to know what our current thinking
about all this is. Since this is in a very early stage of planning, just
about everything is up to debate. To start off the discussion, I’d like
to answer the following questions:</p>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
<p>What parts of RHQ would you like to sync between RHQ servers?</p>
<div class="ulist">
<ul>
<li>
<p>server configuration</p>
</li>
<li>
<p>users</p>
</li>
<li>
<p>roles</p>
</li>
<li>
<p>metric templates</p>
</li>
<li>
<p>alert templates</p>
</li>
<li>
<p>content sources</p>
</li>
<li>
<p>repos</p>
</li>
<li>
<p>packages</p>
</li>
<li>
<p>dyna groups</p>
</li>
<li>
<p>plugins</p>
</li>
<li>
<p>configuration, connection settings of a resource</p>
</li>
<li>
<p>metric schedules of a resource</p>
</li>
<li>
<p>alert definitions of a resource</p>
</li>
</ul>
</div>
</li>
<li>
<p>How granular should the export be?</p>
<div class="ulist">
<ul>
<li>
<p>all or nothing - i.e. "true" sync</p>
</li>
<li>
<p>per "subsystem" (i.e. all users&roles, all templates, content sources
&repos & packages, …)</p>
</li>
<li>
<p>pick and choose individual entities</p>
</li>
</ul>
</div>
</li>
<li>
<p>How segmented should the export be?</p>
<div class="ulist">
<ul>
<li>
<p>lump different entity types together in one export file</p>
</li>
<li>
<p>export per "subsystem"</p>
</li>
</ul>
</div>
</li>
<li>
<p>When should the import be run?</p>
<div class="ulist">
<ul>
<li>
<p>during RHQ server installation</p>
</li>
<li>
<p>any time</p>
</li>
</ul>
</div>
</li>
</ol>
</div>
<div class="paragraph">
<p>If you want to shape the future of RHQ, now’s the time! ;-)</p>
</div>
http://lukas.krejci.pw/posts/2011/03/08/scripted-alert-notifications-in-rhq/Scripted alert notifications in RHQ2014-11-24T23:25:00+01:002011-03-08T00:00:00+00:00
Since RHQ3, we support "alert sender" server plugins. Basically an alert
sender is a piece of code that can generate some sort of response to the
firing of an alert.
There’s a whole bunch of these in RHQ, including:
emails - sending emails to the configured addresses that an alert
occured within the system
roles, users - notifying members of given roles or users about the
alert
mobicents - sends SMS messages about the alert
log4j - writes a log entry when the alert fires
operation - executes an operation on some resource in the RHQ
inventory when the alert fires
…
This blog post is about a new such alert sender that...
<div class="paragraph">
<p>Since RHQ3, we support "alert sender" server plugins. Basically an alert
sender is a piece of code that can generate some sort of response to the
firing of an alert.</p>
</div>
<div class="paragraph">
<p>There’s a whole bunch of these in RHQ, including:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>emails - sending emails to the configured addresses that an alert
occured within the system</p>
</li>
<li>
<p>roles, users - notifying members of given roles or users about the
alert</p>
</li>
<li>
<p>mobicents - sends SMS messages about the alert</p>
</li>
<li>
<p>log4j - writes a log entry when the alert fires</p>
</li>
<li>
<p>operation - executes an operation on some resource in the RHQ
inventory when the alert fires</p>
</li>
<li>
<p>…</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>This blog post is about a new such alert sender that is capable of
executing a CLI script.</p>
</div>
<div class="paragraph">
<p>RHQ has a command-line client, the CLI, which is able to remotely
connect to an RHQ server and execute commands on it. Basically the CLI
enables the users to use the Remote API of the RHQ server in a
Javascript environment.</p>
</div>
<div class="paragraph">
<p>Now with the CLI scripts as the alert notifications you have the same
power at your fingertips as you have in the CLI directly on the server.
The scripts can do literally anything you can do in your CLI scripts.</p>
</div>
<div class="paragraph">
<p>As an example, consider the following script:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="java"><table class="CodeRay"><tr>
<td class="line-numbers"><pre>1
2
3
4
5
6
7
8
9
<strong>10</strong>
11
12
13
14
15
16
17
18
19
<strong>20</strong>
21
22
23
24
25
26
27
28
29
<strong>30</strong>
31
32
33
34
35
36
37
38
39
<strong>40</strong>
41
42
43
44
45
46
47
48
49
<strong>50</strong>
51
52
53
54
55
56
57
58
59
<strong>60</strong>
61
62
63
64</pre></td>
<td class="code"><pre><span class="comment">/*
* This script is supposed to be notifying about alerts on a web application.
* It will save some stats into a file on the RHQ server and then invoke a bash
* script if it finds it necessary.
*/</span>
<span class="comment">//get the proxied resource so that I can use the more convenient syntax than</span>
<span class="comment">//just the raw calls to the remote APIs</span>
<span class="comment">//notice the predefined variable 'alert' that contains the object of the alert that is being</span>
<span class="comment">//fired</span>
var myResource = ProxyFactory.getResource(alert.alertDefinition.resource.id)
<span class="comment">//find the metric (aka measurement) for the "Sessions created per Minute"</span>
<span class="comment">//this will give us the picture about the load on the web app</span>
var definitionCriteria = <span class="keyword">new</span> MeasurementDefinitionCriteria()
definitionCriteria.addFilterDisplayName(<span class="string"><span class="delimiter">'</span><span class="content">Sessions created per Minute</span><span class="delimiter">'</span></span>)
definitionCriteria.addFilterResourceTypeId(myResource.resourceType.id)
var definitions = MeasumentDefinitionManager.findMeasurementDefinitionsByCriteria(definitionCriteria)
<span class="comment">//only continue if we have the definition</span>
<span class="keyword">if</span> (definitions.empty) {
<span class="keyword">throw</span> <span class="keyword">new</span> java.lang.Exception(<span class="string"><span class="delimiter">"</span><span class="content">Could not get 'Sessions created per Minute' metric on resource </span><span class="delimiter">"</span></span>
+ myResource.id)
}
var definition = definitions.get(<span class="integer">0</span>)
<span class="comment">//start date is now - 8hrs</span>
var startDate = <span class="keyword">new</span> <span class="predefined-type">Date</span>() - <span class="integer">8</span> * <span class="integer">3600</span> * <span class="integer">1000</span> <span class="comment">//8 hrs in milliseconds</span>
var endDate = <span class="keyword">new</span> <span class="predefined-type">Date</span>()
<span class="comment">//get the data of the metric for the last 8 hours, chunked up to 60 intervals</span>
var data = MeasurementDataManager.findDataForResource(myResource.id, [ definition.id ], startDate, endDate, <span class="integer">60</span>)
exporter.setTarget(<span class="string"><span class="delimiter">'</span><span class="content">csv</span><span class="delimiter">'</span></span>, <span class="string"><span class="delimiter">'</span><span class="content">/the/output/folder/for/my/metrics/</span><span class="delimiter">'</span></span> + endDate + <span class="string"><span class="delimiter">'</span><span class="content">.csv</span><span class="delimiter">'</span></span>)
<span class="comment">//the data contains an entry for each of the definitions we asked the data for...</span>
exporter.write(data.get(<span class="integer">0</span>))
<span class="comment">//ok, we've exported the stats</span>
<span class="comment">//now we want to make sure that our database is still running</span>
<span class="comment">//let's suppose the resource id of the datasource is "well-known"</span>
<span class="comment">//we could get it using criteria APIs as well, of course</span>
var dataSource = ProxyFactory.getResource(<span class="integer">10411</span>)
<span class="comment">//now check if the datasource's underlying connection is up</span>
<span class="comment">//There is an operation defined on a "Data Source" resource type, which we can call</span>
<span class="comment">//as a simple javascript method on the resource proxy</span>
connectionTest = dataSource.testConnection()
<span class="comment">//the result will be null, if the operation couldn't be invoked at all or if it took</span>
<span class="comment">//too long. Otherwise it will be a configuration object representing the operation</span>
<span class="comment">//results as defined by the operation definition.</span>
<span class="comment">//In this case, the result of an operation is a configuration object with a single</span>
<span class="comment">//property called "result" which is true if the connection could be established and</span>
<span class="comment">//false otherwise</span>
<span class="keyword">if</span> (connectionTest == <span class="predefined-constant">null</span> || connectionTest.get(<span class="string"><span class="delimiter">'</span><span class="content">result</span><span class="delimiter">'</span></span>).booleanValue == <span class="predefined-constant">false</span>) {
<span class="comment">//ok, this means we had problems connecting to the database</span>
<span class="comment">//let's suppose there's an executable bash script somewhere on the server that</span>
<span class="comment">//the admins use to restart the database</span>
java.lang.Runtime.getRuntime().exec(<span class="string"><span class="delimiter">'</span><span class="content">/somewhere/on/the/server/restart-database.sh</span><span class="delimiter">'</span></span>)
}</pre></td>
</tr></table></code></pre>
</div>
</div>
<div class="paragraph">
<p>In another words, it is quite powerful :)</p>
</div>
<div class="paragraph">
<p>There is a design wiki page with documentation of the feature, if you’re
interested in reading more about it:<br>
<a href="http://wiki.rhq-project.org/display/RHQ/Design+-Serverside+scripts%5Bhttp://wiki.rhq-project.org/display/RHQ/Design-+Serverside+scripts" class="bare">http://wiki.rhq-project.org/display/RHQ/Design+-Serverside+scripts[http://wiki.rhq-project.org/display/RHQ/Design-+Serverside+scripts</a>]</p>
</div>
<div class="paragraph">
<p>There’s the brand new <a href="http://rhq-project.org/display/RHQ/Download">RHQ
4.0.0.Beta1</a> out that contains this new feature. Go check it out!</p>
</div>
<div class="paragraph">
<p>For the impatient, I recorded a short screencast of the new feature in
action. <a href="http://vimeo.com/20791639" class="bare">http://vimeo.com/20791639</a></p>
</div>
<div class="paragraph">
<p>It is best viewed in HD but for that you have to view it directly on
vimeo.com. Just click the "HD" in the video.</p>
</div>
http://lukas.krejci.pw/posts/2010/10/06/upgrading-resources-in-rhq/Upgrading Resources In RHQ2014-11-24T23:25:00+01:002010-10-06T00:00:00+00:00
Humans make mistakes. That’s because they learn and by making mistakes
and experimenting they perfect their skills. Computers on the other hand
only do as they are told. When a human interacts with the computer, she
expects the computer to be human-like to the extent that it can recover
or react to the mistakes she makes while learning the rules of
interaction with it.
In case of RHQ one of the users is a plugin
developer. An RHQ (agent) plugin is a "thing" that talks to some other
software and can configure and monitor it. The other type of the RHQ
user is the system administrator that uses RHQ...
<div class="paragraph">
<p>Humans make mistakes. That’s because they learn and by making mistakes
and experimenting they perfect their skills. Computers on the other hand
only do as they are told. When a human interacts with the computer, she
expects the computer to be human-like to the extent that it can recover
or react to the mistakes she makes while learning the rules of
interaction with it.</p>
</div>
<div class="paragraph">
<p>In case of <a href="http://www.rhq-project.org">RHQ</a> one of the users is a plugin
developer. An RHQ (agent) plugin is a "thing" that talks to some other
software and can configure and monitor it. The other type of the RHQ
user is the system administrator that uses RHQ and its plugins to manage
their IT infrastructure. For the administrator, the plugin becomes part
of the "computer". But the plugin is made by humans and humans make
mistakes.</p>
</div>
<div class="paragraph">
<p>One of the mistakes the plugin developer can make is to assign a wrong
"resource key" during the discovery of resources. A resource key is
something that uniquely identifies the particular "resource" the plugin
can talk to. You can dive much deeper into the details of this
<a href="http://rhq-project.org/display/RHQ/Writing+Custom+Plugins#WritingCustomPlugins-Identity">here</a>
but in a nutshell, the resource key is extracted from the data the
plugin can gather about a resource in such a way that if at some later
point in time the plugin is told to rediscover the resources it can
manage, the resource key will remain the same for the same resources.
Usually, the resource key is a file-system location of some significant
configuration file, an installation directory, a port a service is
listening on, a CPU id, a mount point, etc. Whatever fits the need of
the particular plugin.</p>
</div>
<div class="paragraph">
<p>If the plugin developer realized she made a mistake in the way the
resource key is generated and that it for example doesn’t identify the
resource in a completely unique way, the only thing she could do is to
create a new version of the plugin with the fix and distribute it to her
customers. Easy.</p>
</div>
<div class="paragraph">
<p>But the consequences for the customers (i.e. the system administrators)
were quite severe. Because the resource key algorithm changed, the
resources that they already had in their inventories and that they
managed, collected stats on, had alerts defined for, etc. suddenly
became defunct and a new resource (or even more resources, with the new
resource keys) appeared in their place. The administrator would then
have to go and define their alerts, add the new resource(s) to the
groups they defined for them, etc. all over again. Not to mention the
fact the historical data would from now on be split between the "old"
and "new" resource so if the admin wanted to retain the historical data
they could not just delete the defunct resource. They’d have to keep it
in their inventory in which the resource would forever stay in
unavailable state showing red icons signifying a problem where there
wouldn’t be any (well, the problem was on the plugin developer side, but
the admin would suffer for it).</p>
</div>
<div class="paragraph">
<p><strong>But all that changed.</strong> Yesterday, I merged in a new feature that
enables the plugin developers to fix their past mistakes in a way that
the system admins wouldn’t be punished for.</p>
</div>
<div class="paragraph">
<p>I called the process "resource upgrade" because it enables the plugin
developers to change the data of the existing resources to "upgrade"
them so that they conform to the latest version of the plugin code. For
now the feature is quite rudimentary and only enables to upgrade the
resource key, resource name and description. Obviously, the other big
candidates for upgrade would be the plugin configuration (aka connection
settings) and resource configuration. When we were thinking about this
though we realized that not only would implmenenting the configuration
upgrades be quite complex because we would basically have to break the
fundamental principle of RHQ configuration which is that the RHQ server
is the authoritative source of configuration data, but we couldn’t find
a plugin that would benefit from such capability.</p>
</div>
<div class="paragraph">
<p>If you happen to have a custom RHQ plugin for your piece of software and
say to yourself, damn, why oh why didn’t they think about my use case,
I’d love to be able to upgrade the configuration of existing resources,
please leave a comment here. I’d love to hear about your needs so that
we can think about supporting them in the future.</p>
</div>
<div class="paragraph">
<p>When I said that the resource upgrade supports upgrading the resource
name and description I partially lied though. The actual update of these
is guarded by a server configuration setting because both resource name
and description can be updated by the users and we wouldn’t want to
overwrite their updates without consent. For now there is even no way to
enable that setting in the UI even though it’d be trivial to add. The
problem with this is that for this feature to be useful to our current
community and customers, we’d have to update all the plugins with code
that would implement it which becomes a substantial amount of work with
our 30+ plugins at the moment. But if you feel like you absolutely need
this, contact our team. Anything is possible when there’s enough backing
;)</p>
</div>
<div class="paragraph">
<p>You can read more about the technicalities of the implementation on our
<a href="http://rhq-project.org/display/RHQ/Design+-+Resource+Upgrade">wiki</a>.</p>
</div>
http://lukas.krejci.pw/posts/2010/09/20/database-setup-for-testng-tests/Database setup for TestNG tests2014-11-24T23:25:00+01:002010-09-20T00:00:00+00:00
In my
previous
post I talked about the approach I took to export data from a database
using a JPA model. I also mentioned that that was a part of a larger
effort to support performance testing that we are currently implementing
for RHQ. This post is a follow-up on that
theme. This time we’re going to take a look at how to use the exported
data in TestNG based tests.
The problem at hand is basically restoring the database to the exact
state as it was when the data for the test was exported. This gets
non-trivial in an evolving project like RHQ where we constantly change
the DB schema...
<div class="paragraph">
<p>In my
<a href="http://metlos.wordpress.com/2010/09/15/how-to-export-data-from-a-db-using-jpa-model/">previous
post</a> I talked about the approach I took to export data from a database
using a JPA model. I also mentioned that that was a part of a larger
effort to support performance testing that we are currently implementing
for <a href="http://www.rhq-project.org">RHQ</a>. This post is a follow-up on that
theme. This time we’re going to take a look at how to use the exported
data in <a href="http://testng.org">TestNG</a> based tests.</p>
</div>
<div class="paragraph">
<p>The problem at hand is basically restoring the database to the exact
state as it was when the data for the test was exported. This gets
non-trivial in an evolving project like RHQ where we constantly change
the DB schema to either add new features or to do performance
enhancements. Before each test, we therefore need to do the following:</p>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
<p>Recreate the database to the minimum supported version.</p>
</li>
<li>
<p>Upgrade the database schema to the version from which the data for
the test was exported from.</p>
</li>
<li>
<p>Import the test data.</p>
</li>
<li>
<p>Upgrade the schema (now with the correct data) to the latest
database version.</p>
</li>
<li>
<p>Run the test.</p>
</li>
</ol>
</div>
<div class="paragraph">
<p>TestNG is all about annotations so all this should ideally happen
transparently to the test just by annotating the methods somehow. As far
as I know there is no easy way to add a new custom annotation to TestNG
core, but fortunately TestNG 5.12 added support for
<a href="http://testng.org/javadocs/index.html?org/testng/annotations/Listeners.html"><code>@Listeners</code></a>
annotation which can be used to add any TestNG defined listener to the
test. By implementing
<a href="http://testng.org/javadocs/index.html?org/testng/IInvokedMethodListener.html"><code>IInvokedMethodListener</code></a>,
we can check for presence of our new annotations on the tests and thus
effectively implement a new TestNG "managed" annotation.</p>
</div>
<div class="paragraph">
<p>With <code>@Listeners</code> and <code>IInvokedMethodListener</code>, the implementation is
quite easy. We can define a simple annotation that will provide
configuration for restoring the database state to be used on the test
methods and implement the setup in our method listener.</p>
</div>
<div class="paragraph">
<p>Let’s take a look at the actual database state annotation copied from
our code base:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="java"><table class="CodeRay"><tr>
<td class="line-numbers"><pre>1
2
3
4
5
6
7
8
9
<strong>10</strong>
11
12
13
14
15
16
17
18
19
<strong>20</strong>
21
22
23
24
25
26
27
28
29
<strong>30</strong>
31
32
33
34
35
36
37
38
39</pre></td>
<td class="code"><pre><span class="comment">/**
* An annotation to associate a test method with a required state of the database.
*
* @author Lukas Krejci
*/</span>
<span class="annotation">@Retention</span>(value = <span class="predefined-type">RetentionPolicy</span>.RUNTIME)
<span class="annotation">@Target</span>(value = { <span class="predefined-type">ElementType</span>.METHOD })
<span class="directive">public</span> <span class="annotation">@interface</span> DatabaseState {
<span class="comment">/**
* The location of the database state export file.
*/</span>
<span class="predefined-type">String</span> url();
<span class="comment">/**
* The version of the RHQ database the export file is generated from.
* Before the data from the export file are imported into the database, the database
* is freshly created and upgraded to this version. After that, the export file
* is imported to it and the database is then upgraded to the latest version.
*/</span>
<span class="predefined-type">String</span> dbVersion();
<span class="comment">/**
* Where is the export file accessible from (defaults to {@link DatabaseStateStorage#CLASSLOADER}).
*/</span>
DatabaseStateStorage storage() <span class="keyword">default</span> DatabaseStateStorage.CLASSLOADER;
<span class="comment">/**
* The format of the export file (defaults to zipped xml).
*/</span>
FileFormat format() <span class="keyword">default</span> FileFormat.ZIPPED_XML;
<span class="comment">/**
* The name of the method to provide a JDBC connection object.
* If the method is not specified, the value of the {@link JdbcConnectionProviderMethod} annotation
* is used.
*/</span>
<span class="predefined-type">String</span> connectionProviderMethod() <span class="keyword">default</span> <span class="string"><span class="delimiter">"</span><span class="delimiter">"</span></span>;
}</pre></td>
</tr></table></code></pre>
</div>
</div>
<div class="paragraph">
<p>A test class that would use these would look something like this:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="java"><table class="CodeRay"><tr>
<td class="line-numbers"><pre>1
2
3
4
5
6
7
8
9</pre></td>
<td class="code"><pre><span class="annotation">@Listeners</span>(DatabaseSetupInterceptor.class)
<span class="directive">public</span> <span class="type">class</span> <span class="class">MyDbTests</span> {
<span class="annotation">@Test</span>
<span class="annotation">@DatabaseState</span>(url = <span class="string"><span class="delimiter">"</span><span class="content">my-exported-data.xml.zip</span><span class="delimiter">"</span></span>, dbVersion = <span class="string"><span class="delimiter">"</span><span class="content">2.94</span><span class="delimiter">"</span></span>)
<span class="directive">public</span> <span class="type">void</span> test1() {
...
}
}</pre></td>
</tr></table></code></pre>
</div>
</div>
<div class="paragraph">
<p>I think that most of that is pretty self-explanatory. The only thing
that needs explained further is the <code>dbVersion</code> and how we are dealing
with setting up and upgrading the database schema.</p>
</div>
<div class="paragraph">
<p>In RHQ we have been using our home-grown
<a href="http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=tree;f=modules/core/dbutils">dbutils</a>
that use one XML file to store the "current" database schema definitions
and another XML file (db-upgrade.xml) to detail the individual upgrade
steps that evolve the schema (each such step is considered a schema
"version"). The first XML is used for clean installations and the other
is used to upgrade a schema used in previous versions to the current
one. The <code>dbVersion</code> therefore specifies the version from the
<code>db-upgrade.xml</code>.</p>
</div>
<div class="paragraph">
<p>And that’s basically it. You can check the implementation of the
<a href="http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=blob;f=modules/helpers/perftest-support/src/main/java/org/rhq/helpers/perftest/support/testng/DatabaseSetupInterceptor.java;hb=perftest">DatabaseSetupInterceptor</a>
which does exactly the points 1 to 4 mentioned above.</p>
</div>
<div class="paragraph">
<p>As a final, slightly unrelated, note, we are currently thinking about
migrating our own database setup/upgrade tool to
<a href="http://liquibase.org">liquibase</a>. I think that the above approach should
be easily transferable to it by changing the dbVersion attribute to the
liquibase’s changeset id/author/file combo but I’m no expert in
liquibase. If you happen to know liquibase and think otherwise, please
leave a comment here and we’ll get in touch ;)</p>
</div>
<div class="paragraph">
<p>As with the export tool described in the previous post, I tried to
implement this in a way that wouldn’t be tied to RHQ so this could
potentially be used in other projects (well, with this time, you’d
either have to adopt our dbutils or liquibase, but I think even this
could be made configurable).</p>
</div>
http://lukas.krejci.pw/posts/2010/09/15/how-to-export-data-from-a-db-using-jpa-model/How to export data from a DB using JPA model2014-11-24T23:25:00+01:002010-09-15T00:00:00+00:00
In RHQ, we are currently contemplating
implementing a series of automated performance tests. For those tests to
make any sense, we have to provide them with some initial data to work
with.
So the goal is quite simple. Export some defined dataset from an
existing database, store it away and import it back again before a test
is run. Easy. When I started researching the export part of the problem,
I thought there’s bound to be something out there already in existence
that would do the job. And I was right. The
dbUnit project is exactly what I was looking for.
They support extraction of the data from the database...
<div class="paragraph">
<p>In <a href="http://www.rhq-project.org">RHQ</a>, we are currently contemplating
implementing a series of automated performance tests. For those tests to
make any sense, we have to provide them with some initial data to work
with.</p>
</div>
<div class="paragraph">
<p>So the goal is quite simple. Export some defined dataset from an
existing database, store it away and import it back again before a test
is run. Easy. When I started researching the export part of the problem,
I thought there’s bound to be something out there already in existence
that would do the job. And I was right. The
<a href="http://www.dbunit.org">dbUnit</a> project is exactly what I was looking for.
They support extraction of the data from the database and can even
follow the foreign key relationships (in both ways) to export the
necessary data to keep referential integrity. Great.</p>
</div>
<div class="paragraph">
<p>But wait. Our data model isn’t that simple. I certainly want all the
data that my core dataset depends on to be included in the export, but I
also want <strong>some</strong> of the data that depends on my dataset.</p>
</div>
<div class="paragraph">
<p>Ok that didn’t make much sense, so let me introduce a little example to
explain the problems on. First, let’s look at the class diagram, that
will show the relationships between different entities in the model.</p>
</div>
<div class="paragraph">
<p><a href="../../../../../images/class-diagram.png" data-lightbox="1">
<span class="image thumb"><img src="http://lukas.krejci.pw/images/class-diagram.png" alt="Class diagram"></span>
</a></p>
</div>
<div class="paragraph">
<p>These entities are mapped to these tables:</p>
</div>
<div class="paragraph">
<p><a href="../../../../../images/entity-relationship-diagram.png" data-lightbox="2">
<span class="image thumb"><img src="http://lukas.krejci.pw/images/entity-relationship-diagram.png" alt="Entity diagram"></span>
</a></p>
</div>
<div class="paragraph">
<p>Now let’s say I wanted to export all the resources with their
configurations but I’m not interested in the alert definitions.
Obviously this is going to require some kind of configuration.</p>
</div>
<div class="paragraph">
<p>I could stay on the database level and for example create a
configuration where I would specifically state something like "I want
data from this table." or "I’m interested in this table and all its
dependencies but this particular foreign key." and implement a dbUnit
search based on this configuration but I’m a Java developer and even
though I can write my SQL statements and design a (more or less)
reasonable database schema, I certainly don’t love that job. To find out
the relationships between tables, looking at the JPA annotated Java code
is much quicker and more pleasant to me than looking at table and
foreign key definitions.</p>
</div>
<div class="paragraph">
<p>Before I dive into more details let me show you the configuration file
that will achieve the above goal:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="xml"><table class="CodeRay"><tr>
<td class="line-numbers"><pre>1
2
3
4
5
6
7
8
9
<strong>10</strong></pre></td>
<td class="code"><pre><span class="tag"><graph</span> <span class="attribute-name">includeExplicitDependentsImplicitly</span>=<span class="string"><span class="delimiter">"</span><span class="content">true</span><span class="delimiter">"</span></span>
<span class="attribute-name">packagePrefix</span>=<span class="string"><span class="delimiter">"</span><span class="content">org.rhq.core.domain.resource</span><span class="delimiter">"</span></span><span class="tag">></span>
<span class="tag"><entity</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">"</span><span class="content">Resource</span><span class="delimiter">"</span></span> <span class="attribute-name">root</span>=<span class="string"><span class="delimiter">"</span><span class="content">true</span><span class="delimiter">"</span></span><span class="tag">></span>
<span class="tag"><filter></span>SELECT ID FROM RESOURCE WHERE NAME='myResource'<span class="tag"></filter></span>
<span class="tag"><rel</span> <span class="attribute-name">field</span>=<span class="string"><span class="delimiter">"</span><span class="content">configuration</span><span class="delimiter">"</span></span><span class="tag">/></span>
<span class="tag"></entity></span>
<span class="tag"><entity</span> <span class="attribute-name">name</span>=<span class="string"><span class="delimiter">"</span><span class="content">ResourceType</span><span class="delimiter">"</span></span> <span class="attribute-name">includeAllFields</span>=<span class="string"><span class="delimiter">"</span><span class="content">true</span><span class="delimiter">"</span></span><span class="tag">></span>
<span class="tag"><rel</span> <span class="attribute-name">field</span>=<span class="string"><span class="delimiter">"</span><span class="content">resources</span><span class="delimiter">"</span></span> <span class="attribute-name">exclude</span>=<span class="string"><span class="delimiter">"</span><span class="content">true</span><span class="delimiter">"</span></span><span class="tag">/></span>
<span class="tag"></entity></span>
<span class="tag"></graph></span></pre></td>
</tr></table></code></pre>
</div>
</div>
<div class="paragraph">
<p>This is still a bit of a mouthful but at the same time it’s very
powerful. What I’m basically saying there is that I want to export a
resource with the name "myResource" and I only want to include its
configuration in the export (of course the simple properties of the
resource are implicitly exported but configuration is the only
<strong>relationship</strong> that gets exported along with it). Further, I’m telling
the exporter that it’s free to export all the data of the ResourceType
entities my Resource is dependent upon but I don’t want to include the
resources of the ResourceType in the export. This is to prevent the
other resources to "leak" to the export due to the explicit relationship
of the ResourceType and its "child" Resource entities. The mysterious
“includeExplicitDependetsImplicitly” attribute tells the exporter to
include all dependents of the entities it encounters unless configured
otherwise.</p>
</div>
<div class="paragraph">
<p>I want the above configuration to cause the exporter to include the
following in the export (look at the above class diagram to get a better
understanding why I need the below):</p>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
<p>"myResource" resource</p>
</li>
<li>
<p>Its configuration and all its properties</p>
</li>
<li>
<p>The resource type of the resource</p>
</li>
<li>
<p>The configuration definition associated with that resource type</p>
</li>
<li>
<p>All the property definitions of the configuration definition</p>
</li>
</ol>
</div>
<div class="sect1">
<h2 id="details">
<a class="anchor" href="#details"></a>Details</h2>
<div class="sectionbody">
<div class="paragraph">
<p>To achieve the above functionality I needed to create a bridge that
would look at the JPA annotations in my domain layer classes and
translate the relationships expressed there into SQL terms. Once I have
the SQL representation of the domain model relationships I can feed that
into dbUnit and use it to export and import the data as well (I also let
dbUnit figure out the proper insertion order to keep the referential
integrity but more on that later).</p>
</div>
<div class="paragraph">
<p>The code turned out to be fairly simple and basically consists of
creation of an
<a href="http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=blob;f=modules/helpers/perftest-support/src/main/java/org/rhq/helpers/perftest/support/jpa/EntityDependencyGraph.java;hb=perftest">entity
dependency graph</a>, where nodes represent the JPA entities and edges
represent individual relationships (i.e. a directed, cyclic, multiply
connected graph). The JPA annotations contain all the information to
translate the entities and their relationships into the terms of SQL
tables and columns, the translation is only slightly complicated by the
possibility of relation tables (e.g. a relation table to describe a
<code>@ManyToMany</code> relationship) (the code is
<a href="http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=blob;f=modules/helpers/perftest-support/src/main/java/org/rhq/helpers/perftest/support/jpa/mapping/MappingTranslator.java;hb=perftest">here</a>).</p>
</div>
<div class="paragraph">
<p>With the SQL mapping at hand I could start linking the code I had with
the functionality defined in dbUnit. I chose to implement it as a
<a href="http://www.dbunit.org/apidocs/index.html?org/dbunit/dataset/filter/ITableFilter.html">ITableFilter</a>.
By inheriting from
<a href="http://www.dbunit.org/apidocs/index.html?org/dbunit/database/DatabaseSequenceFilter.html">DatabaseSequenceFilter</a>
I got the correct table order in the export for free and by retaining
the insertion order in the sets of the allowed PKs while traversing the
entity dependency graph, I was also able to retain the correct insertion
order even in cases where a table has a foreign key on itself. My
<code>EntityRelationshipFilter</code> can use the above mentioned configuration to
restrict the traversal of the entity dependency graph and therefore
restrict the resulting export (by relying on
<a href="http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=blob;f=modules/helpers/perftest-support/src/main/java/org/rhq/helpers/perftest/support/jpa/ConfigurableDependencyInclusionResolver.java;hb=perftest">an
inclusion resolver</a> to tell it what to do). You can take a look at the
code
<a href="http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=blob;f=modules/helpers/perftest-support/src/main/java/org/rhq/helpers/perftest/support/dbunit/EntityRelationshipFilter.java;hb=perftest">here</a>.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="conclusion">
<a class="anchor" href="#conclusion"></a>Conclusion</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Relying on dbUnit to do the "low-level" data export and import for me, I
could create a "Java developer friendly" data export tool in just a
little bit more than a week’s time. The good thing is that it is
completely generic and so it could be easily used with other projects
than RHQ (of course, more work would be required on the tool in that
case because the translation from JPA to SQL isn’t completely
implemented. For example it’s missing handling the implicit values of
the JPA annotations (e.g. the table name derived from the class name if
the <code>@Table</code> annotation doesn’t explicitly specify a name) and I’m sure
I missed some corner cases in handling the relationships as well. But it
seems to work for RHQ at the moment which means that it’s already quite
capable because our domain model isn’t a trivial one. If there was
interest, I’d be more than happy to help create a standalone
full-featured tool out of this and take it out of the RHQ
<a href="http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=tree;f=modules/helpers/perftest-support;hb=perftest">source
code</a>. You can read even more about the tool on our wiki
<a href="http://wiki.rhq-project.org/display/RHQ/Entity+Driven+Database+Setup">here</a>.</p>
</div>
<hr>
<div class="paragraph">
<p>In the next blog entry, I’ll take a look at the "import and test" part
of the solution, namely on the integration of the database setup and
data import with TestNG.</p>
</div>
</div>
</div>
http://lukas.krejci.pw/posts/2010/05/11/measuring-ui-performance/Measuring UI Performance2014-11-24T23:25:00+01:002010-05-11T00:00:00+00:00
RHQ is not built to be used by thousands of
users at the same time. We rather add features to the page so that it
contains maximum information and context so that the users can make the
right decisions about their infrastructure. But even then we do care
about a responsive and reasonably performing UI (and the system as a
whole of course).
Recently I’ve been tasked with researching theperformance of our UI layer.
Obviously there are a thousand factors influencing the responsiveness of the
UI but for the web app developer,there are only a few s/he can do something
about. Those in a nutshell are CPU and memory...
<div class="paragraph">
<p><a href="http://rhq-project.org">RHQ</a> is not built to be used by thousands of
users at the same time. We rather add features to the page so that it
contains maximum information and context so that the users can make the
right decisions about their infrastructure. But even then we do care
about a responsive and reasonably performing UI (and the system as a
whole of course).</p>
</div>
<div class="paragraph">
<p>Recently I’ve been tasked with researching theperformance of our UI layer.
Obviously there are a thousand factors influencing the responsiveness of the
UI but for the web app developer,there are only a few s/he can do something
about. Those in a nutshell are CPU and memory usage of the web app itself
and efficiency in the communication with whatever back-end the webapp is
using to pull its data from (usually a DB, but in case of RHQ also the
agents). We did quite a bit of testing on database and agent communication
but we lacked the performance data for the UI layer of RHQ. There obviously
are some candidate pages that one might suspect of a need for performance
enhancements but which ones to pick? The first thing to decide was to
actually determine how to measure the performance of the system. One of
the obvious metrics to use is the response time of the HTTP requests.
This would be a good start because it’d give me a basic understanding of
where the problems might lie. I’d have the starting point to start my
search for performance bottlenecks in the URLs that exhibit slow
response times. On the other hand the results could be skewed by
environmental parameters out of my control like network lag and such
like. But since I had the access to the server I wanted to test on, I
could do better by measuring metrics on the server itself. On the server
side I have much broader choice of what and how I want to measure. If I
wanted to I could even insert "probes" into the server to collect stats
that are otherwise unobtainable from outside of the server’s JVM. I
needed to test several areas of UI using one common testing "algorithm".
I needed to simulate a number of users logging in into the UI and
visiting a number of pages from the "same area" (or rather the same page
with different query parameters). This simple scenario would give me the
least performing areas of UI I could then focus on. To summarize, here
is what I was after:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>don’t bother with response times on the client, i can get the same and
more information on the server-side</p>
</li>
<li>
<p>look for memory hogs</p>
</li>
<li>
<p>look for CPU intensive tasks</p>
</li>
<li>
<p>ideally I want to know more than just the URL at which a bottleneck
might be.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Measuring the memory can either be done by asking <code>ps</code> command or the
JVM itself can provide the heap dump or summary. Measuring CPU is best
done just by <code>ps</code>. The JVM can also provide a thread-dump on demand.
Neither heap summary nor CPU usage nor the thread-dump can be collected
from within the JVM in a simple way (if at all) so I couldn’t track each
request directly using some kind of probe (i.e. adding a filter that
would collect the data to the RHQ web application). All I could do was
to track the request times (either by configuring the embedded Tomcat to
produce access log or better by configuring the RHQ server itself to
collect response time information about itself
(<a href="http://wiki.rhq-project.org/display/RHQ/Response+Time+Filter">http://wiki.rhq-project.org/display/RHQ/Response+Time+Filter</a>))
and periodically collect the stats using an external script.</p>
</div>
<div class="sect3">
<h4 id="technologies-used">
<a class="anchor" href="#technologies-used"></a>Technologies Used</h4>
<div class="paragraph">
<p>For generating the load on the server I used
<a href="http://jakarta.apache.org/jmeter/">JMeter</a>. The nice thing about this
tool is that it is easily parametrized by data either on command-line or
in CSV files. Checkout the very simple JMeter script that I used to
generate the load I needed on the server. The script for starting and
stopping the RHQ server and JMeter and collecting the stats on memory
and CPU usage was simply written in Bash. I used an
<a href="http://www.r-project.org/">R</a> script to generate graphs out of the CSV
files that the Bash script produces from the collected data. You can
find all the source code at the end of this blog entry, if you are
interested in trying it out yourself.</p>
</div>
</div>
<div class="sect3">
<h4 id="interpreting-the-results">
<a class="anchor" href="#interpreting-the-results"></a>Interpreting The Results</h4>
<div class="paragraph">
<p>The script collects 3 kinds of statistics. The per-thread CPU usage, the
JVM heap summary and the JVM thread dump. The scripts run N iterations
and collect the data for each stat in each iteration and store it off in
a file. After the script has finished collecting the data, it creates
CSV files from the CPU usage and heap summary files for easier
consumption of that data. Finally, if R is installed, the CSV files are
converted into graphs (yet easier to digest). The JVM thread dump is
collected so that one can get a basic idea about what each of the
threads in the graph has been doing during the iterations (obviously
this is not precise because of the time elapsed between the CPU usage
and thread dump collections). Let’s take a look at an example graph of
the CPU usage.</p>
</div>
<div class="paragraph">
<p><a href="../../../../../images/measuring_ui_performance__cpu_graph_example.png" data-lightbox="1">
<span class="image thumb"><img src="http://lukas.krejci.pw/images/measuring_ui_performance__cpu_graph_example.png" alt="CPU Graph" width="100%"></span>
</a></p>
</div>
<div class="paragraph">
<p>In there, you can see that one of the threads dominates the CPU usage in
the later iterations. This obviously is a sign of a problem. Taking note
of the thread id (in the legend of the graph) and comparing it with a
<code>"tid"</code> of the threas in the thread dumps in various iterations reveals
that that is the VM Thread doing garbage collection. Looking at the heap
summary graph</p>
</div>
<div class="paragraph">
<p><a href="../../../../../images/measuring_ui_performance__heap_graph_example.png" data-lightbox="2">
<span class="image thumb"><img src="http://lukas.krejci.pw/images/measuring_ui_performance__heap_graph_example.png" alt="Heap Graph" width="100%"></span>
</a></p>
</div>
<div class="paragraph">
<p>one can easily observe that the application was consuming just too much
memory and that the GC, even though it tried really hard, couldn’t
handle the load. From that point on, finding the offending code was as
easy as taking a full heap dump before and after the test (using the
jmap tool that comes with JDK) and finding out which classes contributed
the most to the increased memory usage.
<a href="http://www.eclipse.org/mat/">Eclipse Mat</a> is a great tool for such tasks
and finding the code that caused this issue was a breeze.</p>
</div>
</div>
<div class="sect3">
<h4 id="tests-layout">
<a class="anchor" href="#tests-layout"></a>Tests Layout</h4>
<div class="paragraph">
<p>If you read all the way down here, you probably are interested in how is
this all put together and how does the script obtain all that data. You
can find the link to the complete source code at the end of this entry.
The zip file you can download contains the bash scripts necessary to run
the tests along with an example "test suite" containing the JMeter test
file, example input data for it and example tests.</p>
</div>
<div class="ulist">
<ul>
<li>
<p>*testsuite-runner*the bash script that will start the testsuite in the
background</p>
</li>
<li>
<p>*testsuite-run*examines the testsuite directory and spawns the
individual tests inside it</p>
</li>
<li>
<p>*test-run*runs a single test (i.e. starts RHQ server, starts JMeter
with the test file, collects stats, stops JMeter, stops RHQ server and
produces the outputs)</p>
</li>
<li>
<p><strong>example-testsuite</strong> contains the testsuite files</p>
<div class="ulist">
<ul>
<li>
<p><strong>input</strong> a folder containing input data used in the tests. You will
have to modify these files in order to make the tests work with your RHQ
installation.</p>
</li>
<li>
<p><strong>tests</strong> contains the tests directories</p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>The example invocation of the script would like like the following:</p>
</div>
<div class="paragraph">
<p><code>testsuite-runner path/to/rhq/server/install path/to/jmeter/install NUMBER_OF_STATS_COLLECTIONS path/to/the/testsuite/directory</code></p>
</div>
<div class="paragraph">
<p>This command would start the testsuite in the background. For each test
in the testsuite, an RHQ server and would be started, then a JMeter
instance would be fired up with the test file for given testsuite and a
the provided number of stats measurements would be taken in 10 second
intervals. After that the JMeter and RHQ servers woul be taken down and
the next test in the testsuite would be started.</p>
</div>
</div>
<div class="sect3">
<h4 id="future-work">
<a class="anchor" href="#future-work"></a>Future work</h4>
<div class="paragraph">
<p>Obviously, these scripts are just a quick and dirty solution to my
requirements and have much to be added to them to become truly automated
and useful. For starters, the tests do not connect to the RHQ database
which makes them tied to a particular RHQ inventory (at a defined point
in time), because the inputs of the tests hardcode resource ids. The
first enhancement would therefore be to rewrite the scripts in a more
capable (scripting) language and make them database agnostic.</p>
</div>
<hr>
<div class="paragraph">
<p>The source files and an example testsuite can be downloaded from
<a href="http://www.box.net/shared/tqs4g3k3g9">here</a>.</p>
</div>
</div>
http://lukas.krejci.pw/posts/2010/04/09/hello-world/System.println("Hello world!");2014-11-24T23:25:00+01:002010-04-09T00:00:00+00:00
Well, the time has come for me to reinvent the wheel. This is going to
be a blog about stuff that I find interesting as I move along my way of
an open-source developer.
...
<div class="paragraph">
<p>Well, the time has come for me to reinvent the wheel. This is going to
be a blog about stuff that I find interesting as I move along my way of
an open-source developer.</p>
</div>