Ph: 20040425
20090715 Wednesday July 15, 2009

Does Entropy Affect a Running JVM?

For some time my group has been plagued with a very strange problem. We do a lot of calculations on a Java-based compute grid, where the individual grid engines are fairly long lived. Basically these engines sit around all day waiting for calc requests, and as they come in, they're serviced and responses are sent back.

Pretty simple, right?

Except there's a snag. What we've observed is that the engine JVMs seem to slow down over time. Specifically, after about 18 hours a large percentage of our engines seem to process certain pieces of code very, very slowly. The exact time varies, and can be as low as 12 hours or as high as 36, but 18 is roughly the top of the bell curve. The result is that calc requests that might normally take, say, 10 minutes to run now take 100 minutes to run. If you try to reproduce it, it will run in 10 minutes. If you bounce the same grid engine and re-run the calc there, it takes 10 minutes. But leave a bunch of engines up for greater than 12 hours, and random engines will start hitting this perplexing 10x slowdown.

A team of developers have debugged this issue on and off for a year now, and has narrowed the problem down to one small section of code. This is a biggish 300 line method which does a lot of calculation in a tight loop. This tight loop runs great on a wide range of inputs for sometimes, until it hits some time-related mystery condition 12-36 hours after the engine is started, and then subsequently slows down 10x. We run other code on these same engines and they don't hit this slow down. GC, swapping on the box, etc have all been ruled out. The code itself is completely stateless, and we've verified the behavior with constant inputs on a number of machines over time. The code just literally slows down 10x for no obvious reason.

The only idea left to the team is that somehow the code is switching from hot-spot compiled code to interpreted mode. Somehow. Using the:

-XX:CompileCommand=exclude

...switch to disable hotspot compilation on this function seems to roughly emulate the problem for us (e.g. the function then runs about 10x slower), but we're at a loss as to how this could happen. Or if this is really the explanation. Googling this problem and asking various developers has not yielded an answer. Most people say "Your code must just be slow" - which may be true in the general sense but not in this particular case.

The behavior has been replicated on various 1.5 and 1.6 Sun JVM versions. This is all on Linux boxes, running a variety of 2.4.X versions of linux. I believe we've observed it in both 64 bit and 32 bit environments but I'm not 100% sure on that last.

So - has anyone seen this sort of performance bit-rot in a long-lived JVM process? Has anyone heard of code down-shifting from compiled mode into interpreted model spontaneously? Does entropy affect even software?! Do JVMs dream of byte code sheep?

Posted by Mike Spille in Java at 20090715 Comments[7]

Emerging from the cave Last blog post ever, to pyrasun.com, was 3 years, 356 days ago.

Last post here, to jroller, was even longer - 5 years, one month, 1 day.

Er, in these new futuristic times of twittering and booking and spacing, people still blog, right?

Right?

Let's find out...
Posted by Mike Spille in General at 20090715

20040613 Sunday June 13, 2004

The Number You Have Dialed is No Longer in Service This JRoller weblog page is now officially closed for business. "Pyrasun - The Spille Blog" has been put into permanent stasis, and barring a miracle of epic proportions it shall remain that way until the heat death of the universe (or JRoller shuts it down, whichever comes first). In less cryptic terms the posts here will stay here, and the links should work so long as the JRoller guys decide to leave this stuff on the system. But don't expect anything new to show up at this address.

However, for those always clamoring for more, more, more, all is not lost! Out of the ashes of these bits, a new Blog has been born: Pyrasun 2.0. If you're one of the 7 or so crazies who can endure my writing, head on over to Pyrasun 2.0 where I'll be continuuing my blog in its new and (hopefully) improved form.

Despite the vitriol I've poured out in the past weeks, I'd still like to thank everyone behind JRoller. JRoller is where I got my blogging start, and if JRoller didn't exist there's a good chance that I might never have given blogging a go. As a training ground it was invaluable, and for that I thank everyone involved. I sincerely wish you guys luck, and I hope you can straighten out the current JRoller problems, and perhaps in the future make upgrades slightly less traumatic in nature. If you do it will be a win for everyone.
Posted by Mike Spille in General at 20040613 Comments[0]

20040608 Tuesday June 08, 2004

Somewhere Reliable? Well, once again JRoller was either flat out down, or flakey as hell for most of the day. While it was fun to watch the font size on the front page fluctuate, or show Velocity macros occasionally rather than content, this is getting real old, real fast. I counted 10 times I tried to do something with JRoller today which gave me weird beavior, exceptions, or the site just being plain dead, and about 3 attempts that actually did what they were supposed to. This is just intolerable.

I think it's time to move - I like to write blog entries so people will read them, but JRoller seems hell bent on ensuring that nobody can do that on any sort of reliable basis. Why should I write on a blogging system where readers will be frustrated and unable to read any of my glowing prose (insert :-) here) half the time?

This, of course, leads me to the question of where to go and what to do. I admit, I'm clueless in this genre. I can see that I can either look for another free service, or say screw it and host a blog system on my site, but I'm not sure which alternative would be better. My current personal site lets me do PHP/Perl/mySql stuff, sadly no Java. But I'm not sure whether I should bite into that can of worms (eww!) or hope that another free site might fit the bill. Any suggestions in this regard would be very, very welcome. I do know that I can't tolerate JRoller any longer - I have no interest in using a site that I can't post to, or have other people read! Perhaps this will shock and dismay Rick Ross, and he'll shame me for my use of such emotionally charged obscenities such as "reliable", but if so, so be it. I'd rather rankle the feathers of of the Roller volunteers and move onto a system that's actually (gasp!) usable, thank you very much.

Posted by Mike Spille in General at 20040608 Comments[14]

20040531 Monday May 31, 2004

Jroller Upgrade I have to agree with Hani that the JRoller upgrade is extremely disappointing. I can see tons of work went into the release, but at the same time it looks to me like the developers were tackling things that seemed neat/cool/different and had little interest in improving the usability of the site.

As Hani noted, there's bizarre entries under "Format entry with plugins" when you're writing a new entry, and I haven't a clue what a "Textile Formatter" is - I do know that Textiles are the technical term for various fabrics and their manufacture, but somehow I don't think this is what the developers were getting at. The look has obviously changed - it's not IMHO any better, it's just changed. There's lots of things I'll likely never use, like the ability to apparently delete referrers, and of course the Bookmarks thing I never understood or cared about. And now we can enable/disable comments per entry, and put a time limit on them - but the default doesn't match the global per-blog default, so those who want comments enabled by default have to remember to click it for every damn post they do.

Some old bugs are still there. I still occasionally get told that I'm not authorized for some request. Othertimes I get stack traces that disappear on a reload. I can no longer sort the referrer list. My custom "Archive" and "Comments" pages are now broken. The only good news is that when I log in I no longer get redirected to my CSS file. And there's still no way to look at the directory of users and blogs.

Of course, the biggy is the front page. It's a disaster. The arrangement looks bad. The main recent entries are huge and waste space in every possible way. The "Hot Blogs" is hanging ugly and forlorn off on the right. The JRoller blurb is hanging even uglier in the top middle. I don't think I can convey how bad it looks, and how hard it is to use, using mere words.

Look, I know JRoller is free. But using this "free" service feels more and more expensive in terms of my time. As a developer I know it's fun to tack on new features to a project that are cool and interesting to me - but what I find interesting as the developer rarely seems to match what a user wants. I've been known to scoff at features people have requested in my own work, and then finally caved in and disgustedly took the half hour it took to give the users the stupid feature they wanted - and felt my jaw hit the floor when the users cheered. And meanwhile the super cool fine grained search and sorting feature I spent 3 weeks developing has never been used by a single person.

My advice to the JRoller people, and Roller developers. Focus on usability. Focus on stability. In my own blogging experience, there are a few work horse things I do on a regular basis. Find out what those things are that are universal, and make them painless and pretty. Make the bland process of blogging and reading blogs so easy that people don't even think about it - they just do it. And stamp out the bugs!

On the front page - if you have problems with UI design, find another successful blogging site that bloggers and users alike appreciate, and shamelessly copy their look and feel. Realize that no more than 1 or two short phrases on any given page should be taller than 1/2 inch on a 19" monitor like mine! Put your mind into the shoes of your average blog reader - what would someone not familiar with Roller want to do if they hit the home page? What do real people want to do? For one, think hard on things like a selectably sortable blog directory - it's blindingly obvious that's needed, and yet still I see an juicy entry scroll off one of the front page lists and it's lost forever to me (no, like everyone else, I won't go near search even if I was under torture).

Forget the fancy features. Find out what people use JRoller for day to day and optimize _just that experience_. There's a lot of good in JRoller, but the lack of polish and usability is slowly killing it.

Posted by Mike Spille in General at 20040531 Comments[3]

20040526 Wednesday May 26, 2004

Saying Goodbye To An Old Friend Communities, be they obvious on-line ones like TheServerSide.com, looser confederations like /., or even good old "meat space" tools like newspapers, have always been problematic. Somewhere some community is undergoing some sort of upheaval, be it the fake article stuff that happened at the NYTimes, or the daily goings on in the free-for-all environment of juggernauts like slashdot, or petty scandals on any one of hundreds of online communities. I suppose this just demonstrates the human condition - there's always people looking to cut corners, others on the look out for corner-cutters, individuals and companies out to make a quick buck, and general gadflies flitting about gleefully feeding off of all the controversy where ever they find it. In any community, there'll be rare shining moments of integrity, equally rare shocking scandals, and lots of time in between when plain old information, news, and opinion are exchanged. For good or bad, this is how large groups behave, and their communities reflect that.

I've been involved in a few communities in my time, most recently (and publically) the community over at TheServerSide.com. It's where many people first got to know me, it's the first place that I learned a huge amount about Java technology that I would probably never have stumbled upon on my own, and it's been the battleground where many a flamewar has gone through their inevitable and timeless cycles (and not a few of which I've been involved in, occasionally even being a central figure).

Well, today I say goodbye to that old friend, that community which has figured so prominently in my life for the past couple of years. There's still a great deal of value in that community, but I believe they've let myself and the community down in a way that I can't get past and will not condone, either explicitly or tacitly.

I'm sure a few of you have heard about a controversy that's erupted in the Java space in the past few weeks, perhaps even read about one small portion of it in this blog. TheServerSide.com was central to the whole thing. That's where it started, that's where the most blatant examples were found, and that's where most of the "evidence" in the scandal originated. And, naturally, this was the ultimate place where many people would turn for closure. After all, it's their logs and their people who tracked down who was who and coming from where.

But, as it turns out, TheServerSide.com was glaringly silent through most of this debacle. With stories going out first on blogs, then on slashdot, and later on various other on-line outlets like eWeek, JDJ, and JavaLobby, TheServerSide, one of the two central entities in this controversy, had almost naught to say. A small post was made announcing a new feature for showing users who had posted from a common IP address (with nary a mention as to why this feature was implemented), and an individual from senior management at The MiddleWare Company (TheServerSide's parent company) was quoted in a few sentences in an eWeek article. This was the sum total of their public say on this issue - a generic post on their forums announcing a new "feature", and a couple of blurbs in an eWeek article. Blurbs which, incidentally, labelled the incident a "witch hunt", generally spread tons of FUD, and conveyed no insightful information.

The quotes, for those who haven't read the article:

"Despite the concern that this practice is potentially widespread, it is curious, but disappointing, that this has resulted in a witch hunt specifically directed at only at JBoss Inc.," he said.

"Other organizations could learn from this, too. JBoss becomes an easy and colorful target, not only because they take flamboyant and controversial positions but also because there is a vocal subculture that dislikes JBoss and occasionally thrives on conspiracy theories."
So, the owner of the community from which the entire scandal erupted - the people who's logs proved precisely what was going on, and by whom - calls the offender "an easy and colorful target", and "flamboyant and controversial". Meanwhile, the people who helped tracked this behavior down, and expose it, are engaged in "a witch hunt", and I suppose we compose "a vocal subculture that dislikes JBoss and occasionally thrives on conspiracy theories".

Myself and others had, for months, been begging TheServerSide.com to release the information they had on the whole "anonymous fakers" thing. Way back in February and March of this year, they had gathered enough information to know exactly what was going on, and by whom. One company had been conclusively fingered. The TMC called it "this practice [which] is potentially widespread" - well, if it's so widespread, they have the raw data, the logs, and the analysis tools to find out either way. I've yet to hear of any additional culprits, so my only conclusion is the "widespread" thing is more FUD to make a certain company look not quite so bad in the public eye.

As it is, TheServerSide.com just sat on their information for months. One company was conslusively nailed to the wall. They knew exactly what was going on, and did nothing publically about it. When the story broke - based on information from their logs - they called the people breaking it "a vocal subculture that dislikes JBoss and occasionally thrives on conspiracy theories". Myself and others called on TheServerSide.com to release their information, in suitably protected form, to prove the allegations. We were met with silence.

I talked to a senior The Middleware Company executive about this directly last week, with no tangible result.

The overall effect was that people who had put their reputations on the line to expose this behavior were hung out to dry by TheServerSide and The Middleware Company. People on slashdot and elsewhere who bothered to check the evidence in depth noted that the evidence was slim and hardly conclusive - and they were right. The people who wrote about this story did so based on information we had obtained from TheServerSide - but they have refused to release that information to the public in any form. The result was that the community was left hanging in the breeze. People who dug through the evidence were correct that it was very suspect and incomplete. And the reason why is simple: for myself, I was looking to TSS to fill in the blanks. And they didn't. This left a giant hole for JBoss and their supports to sail through - with no conclusive evidence, they were able to release their PR "response" and get away surprisingly clean-smelling given what they'd been engaged in. To put it plainly, many people were looking to TSS to bring in the final conclusive bits of information - and they bailed on us. If they wished to implicate other individuals or corporations, hey - knock yerself out. If you have evicence that this practice is in fact widespread, bring it on. But no one at TSS or TMC have uttered one whiff of evidence in that direction. And in the case where the evidence is overwhelming, they've sat on it. For those who were wondering, this was the only reason that I published the e-mails from the JBoss team. I trusted TMC to do the right thing and publish their evidence, but they bailed, and the entire issue was left in a murky cloud of doubt that was fooling too many people. When all was said and done, the only conclusive evidence that could be publically put forth were the e-mails I had. I could let it lie, and let both JBoss and TMC pull one over on the community, or publish them (and take the credibility hit). In the end, as you know, I chose the latter.

My own speculation - and this is only speculation - is that JBoss Inc. is a major advertiser on TheServerSide, and went so far as to sponsor the recent TheServerSide Symposium in Las Vegas. In other words, substantial cash was involved. In the end, TheServerSide had a choice: choose between an advertiser and the integrity of their communities. It's obvious which side triumphed - a TMC executive publically calls JBoss "colorful", "flamboyant", and "controversial", and people such as myself are labelled as conspiracy nuts. Let me repeat, this is only speculation - but this speculation makes all the puzzle pieces fit very, very neatly together.

This brings us full circle with my intro into this article. Thanks to human nature, communities of any significant size are problematic. One of those problems, which "publishing"-oriented communities has to constantly deal with, is the issue of integrity vs. revenue. It's the classic problem: what do you do when one of the companies who gives you a great deal of cash is involved in a scandal? Worse - what do you do if your community itself is central to that scandal? Can a community keep its integrity? Possibly more importantly, can it keep its integrity and still make money? Well, we know where TMC and TSS stand on this issue.

As I've mentioned, overall I like TheServerSide.com. Over the years, it's been good to me and proved to be an invaluable resource. My ties to it extended to the point that I was an inaugural member of their Blogger program, and was in negotiations to become a regular columnist. My own articles which they have linked and/or published directly have undoubtedly driven some business their way and netted them some ad dollars.

But when push came to shove, TSS and TMC showed their true colors, and showed to me that sponsors and ad dollars win out. Not just for me - but for the whole community. They knew for months what was going on in their communities in detail, and refused to act. A TMC executive claims that this behavior is "widespread", but shows zero interest in its negative impact in his communities that his company owns, and zero interest in instituting any sort of controls to try to keep this behavior at least somewhat in check.

I'm sure a number of individuals will read this and label me as hopelessly naive, a hopeless romantic, an idiot, or some combination of all the above. The romantic label may apply, but not the rest. I understand ad dollars, I understand what should be taken with a grain of salt, I'm aware of the constant tension between integrity and revenue. And I know that you can't stamp out fake aliases, anonymous abusers, and pressure from advertisers.

I understand all that - but I also understand that a company can at least try to do the right thing. You can't eradicate bad behavior - but you can act to fight against it, to make it harder, to make the people who are cutting corners have to at least break at sweat at being dishonest. The world's a dirty place, and always will be, but that does not mean that we must accept it, and it does not mean we shouldn't at least make the attempt to make our little corners a bit cleaner. If you at least try to inject ethics in your own backyard, there's hope that you might make things better. If you don't even try, you're just guaranteeing that your own little backyard garden is going to be choked with weeds and garbage. Tacit approval of a behavior through inaction has the same consequence as siding with unethical or disruptive behavior.

This dovetails into the recent article by Cameron Purdy and Rickard Oberg on what's happening to our communities. Our communities are being overrun, and it's time to take a stand and start working to eliminate the most disruptive elements. But we need the owners and leaders of the communities first to make this stand and commit to it. Sadly, TSS and TMC appear to have no interest in upholding the integrity of their communities, and rooting out worst kind of behavior when it is blatant and obvious. In my mind, they prefer to take the easy way out and protect advertisers, and the community be damned. I hope they change their mind and see that they can have integrity and still find a way to make money, but that's not going to happen anytime soon.

For this reason I choose to no longer take part in TheServerSide.com, or any other community sponsored by The Middleware Company. I additionally ask that TheServerSide or any other TMC sponsored entity no longer link to my blog or other writings on-line - I cannot enforce this but I can make it clear that it's not in accordance with my wishes. I'm not real happy about this, but I cannot in good conscience continue to take part in a community which knowingly condones such negative and disruptive behavior. Maybe I am a hopeless romantic, but I will not sacrifice a community in pursuit of the allmighty dollar or inflation of my ego - ego and dollars are too easy to come by, and integrity too rare and precious a thing in this universe.

In closing I would like to thank Dion Almaer of TheServerSide. Dion has stood by my side more than once, has offered me a hand when he was under no compulsion to do so, and has clearly fought against policies that are impressed upon him from above. Dion, you're one of the good guys, and none of the negative implications in this article are meant to apply to you. I wish you all the best, and if anyone can bring things back around and salvage your community, I know you can.
Posted by Mike Spille in General at 20040526 Comments[26]
20040524 Monday May 24, 2004

JBoss Employees Admit Guilt The story of JBoss posting under fake aliases on the net, and defaming competitors and critics using those fake names, isn't quite over yet. JBoss Inc. has issued an official "response" to the allegations, a response that does not directly admit any guilt, but states "Astroturfing is hereby banned at JBoss, starting with me".

Many people have read this response, and come to the conclusion that once you lift all the PR speak from the blog entry, Fleury is making a de-facto admission that JBoss was in fact involved in this practice. But in some circles, people still believe JBoss to be an innocent party, and are stating that there still exists no real proof that JBoss has done anything that they've been accused of. To use the words slung about in a few quarters, JBoss Inc is the poor victim of a witch-hunt.

The intent of this entry is to completely debunk this notion by showing e-mails I have received from JBoss employees. I have had several e-mail conversations with JBoss people, in both the past and the present, on this topic of false postings as well as other "discussions" (I use this term loosely - you'll see why in a moment). The current e-mails put any doubts of guilt to rest - they admit it, folks, in no uncertain terms. The past e-mails show two things: what they knew prior to some of their defaming and libelous posts, and the pattern of abuse they fire from their e-mail clients at anyone who criticizes their words, their work, or their company. These e-mails show exactly the type of people we're dealing with, how they think, how they treat others, and certainly put the fake posting scandal to rest.

With a couple of exceptions, these e-mails are completely unedited. Where editing occurs, I have deleted text with the marker [[Deleted MWS]]. The only editing will be via deletion. The reason for these deletions? To avoid propogating unsubstantiated accusations by JBoss people targetted at others. I've left in all the vitriol directed at me, but I have no wish to publish questionable accusations that JBoss has made of other people.

I was intending on showing these e-mails in chronological order, to show the build up to where we are today, but I realized after trying this that it will obscure the issue for many people rather than enlighten. So, instead, I'll begin near the end:

Return-Path: 
Delivered-To: mike@krisnmik.user
Received: (qmail 28379 invoked from network); 21 May 2004 15:44:07 -0000
Received: from unknown (HELO iss04.interliant.com) (198.3.182.194)
  by server256.com with SMTP; 21 May 2004 15:44:07 -0000
Received: from EX-008.mail.navisite.com (ex-008.interliant.com [198.3.179.39])
        by iss04.interliant.com (8.10.2/8.10.2) with ESMTP id i4LFhMV24844
        for ; Fri, 21 May 2004 10:43:22 -0500 (CDT)
Received: from EX-001.mail.navisite.com ([172.16.1.11]) by EX-008.mail.navisite.com with Microsoft SMTPSVC(6.0.3790.0);
         Fri, 21 May 2004 10:41:06 -0500
X-MimeOLE: Produced By Microsoft Exchange V6.5.6944.0
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/related;
        type="multipart/alternative";
        boundary="----_=_NextPart_001_01C43F4A.04744D86"
Subject: Embarrassed and truly sorry
Date: Fri, 21 May 2004 10:43:07 -0500
Message-ID: <75653EEEF6218646B5239EA1D5204E26027D8F2A@EX-001.mail.navisite.com>
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator: 
Thread-Topic: Embarrassed and truly sorry
Thread-Index: AcQ/Sk/z5jtb8HHXQSuKPEb2RIHenA==
From: "Ben  Sabrin" 
To: 
X-OriginalArrivalTime: 21 May 2004 15:41:06.0284 (UTC) FILETIME=[07B7EAC0:01C43F4A]

Dear Mike, 
 
I want to write you a quick personal note and hope that this stays between me and you.  I am extremely ashamed and
embarrassed that I stooped to such a low level in trying to erode your professional credibility.  This was childish
and just plain wrong and for this I am sorry.  I also want to make something very clear, this was not at the direction 
of my employer nor was this a corporate strategy.  And nobody is making me write this letter except me, myself and I.
I am a big boy and boys can stand up for their actions.  I do regret the behavior I have displayed to you.  
 
As for the attacking of the quality of the CDN guys, I have no remorse for this.  I have roomed with Dain Sundstrom and
traveled with him to several conferences and client meetings.  Talk about unprofessional behavior, what do you describe
resigning for from your current employer through a press release?  We made reservations for them to Java One, which is
where they planned their coming out party.  I have no regret for anything I said about any of them, they plotted for 6
months their dismissal from JBoss while getting paid.  I talked to Dain a lot and even used to like him a lot.  He even
went out of his way to recommend we hire Jeremy Boynes in February, when the plot for CDN was well under way.  So while
this does not justify my behavior at all, I hope you can understand why the bad blood is there.  
 
So, if I have harmed you, I am sorry.  I wrote that stupidity on a Saturday from my home and actually got my ass chewed
out pretty bad internally.  I would be more than happy to meet you face to face for us each to prove to each other that
we are not bad people.  
 
Regards,
 
 
Ben Sabrin
Director, Sales North America
404-467-8555 x202
404-664-9466 mobile
404-948-1496 fax
ben@jboss.com
The above e-mail was sent by Ben Sabrin, as a response to an e-mail I sent to Bill Burke and Ben Sabrin. Ben Sabrin is the listed on the JBoss Inc. web site as the Director of Sales and Business Developer. In his e-mail, he signs it merely as "Director, Sales North America". Bill Burke is the Chief Software Architect at JBoss.

This is the original e-mail I sent to these two gentlemen:
To: Bill Burke ,ben@jboss.org
Subject: Any comment?


So Bill, any comment to the blogs that have circulated around in the past 24 hours?

How 'bout you, Ben?  You wrote an e-mail to me in September of 2003 saying "Being in business development I
am always interested in why people feel the way they do and how problems like this can be avoided".
Now it looks to me like you you, as Arun Patel, were the guy trying to ruin my professional reputation.

Are yourselves and JBoss Inc. going to comment?

        -Mike
This e-mail was sent by me to Burke and Sabrin after the blog entries by myself, Cameron, Rickard, and Hani, but before the ./ article was posted. FYI, the e-mail I'm referring to is the following which I received from Ben Sabrin way back in September of 2003:
Return-Path: 
Delivered-To: mike@krisnmik.user
Received: (qmail 8960 invoked from network); 18 Sep 2003 03:58:06 -0000
Received: from unknown (HELO imf19aec.mail.bellsouth.net) (205.152.59.67)
  by server256.com with SMTP; 18 Sep 2003 03:58:06 -0000
Received: from VALUED03BD67F4 ([65.80.83.108])
          by imf19aec.mail.bellsouth.net
          (InterMail vM.5.01.05.27 201-253-122-126-127-20021220) with SMTP
          id <20030918035732.EPFX24929.imf19aec.mail.bellsouth.net@VALUED03BD67F4>
          for ; Wed, 17 Sep 2003 23:57:32 -0400
Reply-To: 
From: "Ben Sabrin" 
To: 
Subject: Why all the hate?
Date: Wed, 17 Sep 2003 23:57:36 -0400
Message-ID: 
MIME-Version: 1.0
Content-Type: multipart/related;
        boundary="----=_NextPart_000_0690_01C37D77.77DDA320"
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.6604 (9.0.2911.0)
Importance: Normal
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1165

Mike, 
 
Just curious as to where all of your hatred for JBoss and JBoss Group has come from.  Have any of the people
who work for JBoss, especially Marc Fleury ever done anything to you personally?  I am just curious, as
you seem to really hate what we are doing.  Being in business development I am always interested in why
people feel the way they do and how problems like this can be avoided.  
 
I read through your site and you seem like a good person.  Perhaps you and I can chat? 
 
Regards,
 
[]
Professional Open Source
Ben Sabrin
Director of Sales and Business Development
404-467-8555 x202
ben@jboss.org
This was my response to Ben's e-mail:
From :  mike        
To :     (Add to Addressbook)    
Subject :       Re: Why all the hate?   
Date :  18 Sep 2003 11:01:30

Thanks for the note, it was interesting to hear from you. 

I wouldn't call the issue one of hate for me, but rather one of frustration. On the Fleury side,
it's frustrating to see JBoss snapping up some of the best Open Source talent there is and redirecting
their efforts away from their primary claims to fame and towards more JBossian things. And Mr. Fleury's
double-speak doesn't help very much. 

The above, however, is relatively minor. I understand business, and I understand personalities such
as Mr. Fleury's, having worked for them in the past. I have a hard time getting worked up over their theatrics
now adays. 

The more urgent issue for me is that of Mr. Burke. In attempting to figure out how JBoss actually works, I've
had online discussions with him about JBoss AOP, the new JBoss JMS, and a few other ancillary topics. In
every case he was not only arrogant, but mis-stated what the code did and its capabilities. I can deal with
arrogance from people who are right - they earn it - I cannot deal with arrogance from people who are consistently
wrong. 

I have gone to the effort to actually document the behavior of what piece of JBoss AOP, and pointed out how
different the implementation was from his descriptions online. I also pointed out that Bela Ban and others
disagreed with his assertions. The end result is more arrogance, more strutting about. 

Frankly, Mr. Burke is giving JBoss a rather bad name in many circles. He goes around bad mouthing CDN people
constantly, gets obviously upset if someone points out errors in his technical arguments, and makes incorrect
and pretty negative accusations about people without substance behind them (such as claiming that I was a dupe
working for a competitor in disguise on TSS). 

What I completely fail to understand is how someone like that is "Chief Architect", and talented people with
integrity and polish like Bela Ban and Gavin King have to play second fiddle to him. It makes you just want to
give up on JBoss in disgust and go elsewhere. 

For what it's worth I work for a subsidiary of the New York Stock Exchange - please don't make this public, as NYSE
and SIAC are very sensitive about divulging technologies they are using or investigating. Mr. Burke's comments, as
Chief Architect of JBoss Group, has seriously damaged consideration of using JBoss in expanded roles in this
environment. 

-Mike
Yeah, you can see from this that Mr. Burke and myself have a not-so-pleasant history. My primary point was the final paragraph of this e-mail:
For what it's worth I work for a subsidiary of the New York Stock Exchange - please don't make this public, as NYSE and SIAC are very sensitive about divulging technologies they are using or investigating. Mr. Burke's comments, as Chief Architect of JBoss Group, has seriously damaged consideration of using JBoss in expanded roles in this environment.
Notice the "please don't make this public" part. If you recall, Ben as "Arun Patel" purposefully pursued me in a TSS thread talking about my place of employment, and actually going so far as to claim I didn't work that. Ben's purpose there was obviously to discredit me, and in all likelihood he was trying to get me in trouble with my employers as well (which is ironic, since he claimed I didn't work for them :-).

The following are Bill Burke's comments to my "Any Comment" e-mail sent on the 18th of May. Some of these e-mails have been edited as I described at the introduction to this blog entry:
Return-Path: 
Delivered-To: mike@krisnmik.user
Received: (qmail 27237 invoked from network); 18 May 2004 13:07:37 -0000
Received: from unknown (HELO iss05.interliant.com) (198.3.182.195)
  by server256.com with SMTP; 18 May 2004 13:07:37 -0000
Received: from EX-008.mail.navisite.com (ex-008.interliant.com [198.3.179.39])
        by iss05.interliant.com (8.10.2/8.10.2) with ESMTP id i4ID6rX08221
        for ; Tue, 18 May 2004 08:06:53 -0500 (CDT)
Received: from jboss.org ([24.128.152.56]) by EX-008.mail.navisite.com with Microsoft SMTPSVC(6.0.3790.0);
         Tue, 18 May 2004 08:04:37 -0500
Message-ID: <40AA0A64.6030300@jboss.org>
Date: Tue, 18 May 2004 09:06:44 -0400
From: Bill Burke 
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.4) Gecko/20030624 Netscape/7.1 (ax)
X-Accept-Language: en-us, en
MIME-Version: 1.0
To: Mike Spille 
Subject: Re: Any comment?
References: <6.0.3.0.2.20040518011034.02008008@mail.krisnmike.com>
In-Reply-To: <6.0.3.0.2.20040518011034.02008008@mail.krisnmike.com>
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
X-OriginalArrivalTime: 18 May 2004 13:04:37.0870 (UTC) FILETIME=[AC8C60E0:01C43CD8]

Joe Murray actually made correct, honest, and factual comments.  Go reread the posts.  He was
obviously trying to debunk obvious FUD from Mike Spille as it was obvious that Mike Spille didn't
know what he was talking about.  I'm glad Joe Murray stepped up to say what I wanted to say as I was banned
from posting on TSS at the time.

As far as Arun Patel, you'll have to take up any issues you have with him as I had nothing to do with it.

Bill

P.S. As far as ruining professional reputations, you should turn an eye towards your own posts to see how
often you attempt such a practice.

Mike Spille wrote:
So Bill, any comment to the blogs that have circulated around in the past 24 hours?
How 'bout you, Ben?  You wrote an e-mail to me in September of 2003 saying "Being in business
development I am always interested in why people feel the way they do and how problems like this can be
avoided".  

Now it looks to me like you you, as Arun Patel, were the guy trying to ruin my professional reputation.
Are yourselves and JBoss Inc. going to comment?
        -Mike

The above was shortly followed by this:
Return-Path: 
Delivered-To: mike@krisnmik.user
Received: (qmail 5604 invoked from network); 18 May 2004 13:39:20 -0000
Received: from unknown (HELO iss05.interliant.com) (198.3.182.195)
  by server256.com with SMTP; 18 May 2004 13:39:20 -0000
Received: from EX-008.mail.navisite.com (ex-008.interliant.com [198.3.179.39])
        by iss05.interliant.com (8.10.2/8.10.2) with ESMTP id i4IDcaX17755
        for ; Tue, 18 May 2004 08:38:36 -0500 (CDT)
Received: from jboss.org ([24.128.152.56]) by EX-008.mail.navisite.com with Microsoft SMTPSVC(6.0.3790.0);
         Tue, 18 May 2004 08:36:21 -0500
Message-ID: <40AA11D8.6050106@jboss.org>
Date: Tue, 18 May 2004 09:38:32 -0400
From: Bill Burke 
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.4) Gecko/20030624 Netscape/7.1 (ax)
X-Accept-Language: en-us, en
MIME-Version: 1.0
To: Mike Spille 
Subject: Re: Any comment?
References: <6.0.3.0.2.20040518011034.02008008@mail.krisnmike.com> <40AA0A64.6030300@jboss.org>
In-Reply-To: <40AA0A64.6030300@jboss.org>
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
X-OriginalArrivalTime: 18 May 2004 13:36:21.0885 (UTC) FILETIME=[1B6E02D0:01C43CDD]

Wanted to add one more thing about Joe Murray's comments as I read your blog more.

He was correct in saying that CMP and JMS performance had improved dramatically starting with
the 3.2.2 series as we had actually put a lot of work into performance and benchmarking starting
with that release.  I had actually written a blog about it months ago right after the 3.2.2
release.

He was also correct in his other statements on what effort was being put where in JBoss development
by JBoss committers over the past year.  I should know as I pay attention to JBoss CVS commits
even more than you do.

Again, it seemed you were trying to trash JBoss's, the project's, reputation, and I'm glad Joe
Murray correctly called on your very incorrect statements.



Bill
I responded to that e-mail. The next item is my response with Bill's comments interspersed (his e-mail fully enfolded mine, so in the interests of space it makes sense to me to just show his quoted-response to my e-mail).
Return-Path: 
Delivered-To: mike@krisnmik.user
Received: (qmail 31485 invoked from network); 18 May 2004 16:37:30 -0000
Received: from unknown (HELO iss05.interliant.com) (198.3.182.195)
  by server256.com with SMTP; 18 May 2004 16:37:30 -0000
Received: from EX-008.mail.navisite.com (ex-008.interliant.com [198.3.179.39])
        by iss05.interliant.com (8.10.2/8.10.2) with ESMTP id i4IGalX00426
        for ; Tue, 18 May 2004 11:36:47 -0500 (CDT)
Received: from jboss.org ([24.128.152.56]) by EX-008.mail.navisite.com with Microsoft SMTPSVC(6.0.3790.0);
         Tue, 18 May 2004 11:30:18 -0500
Message-ID: <40AA3A9A.2020901@jboss.org>
Date: Tue, 18 May 2004 12:32:26 -0400
From: Bill Burke 
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.4) Gecko/20030624 Netscape/7.1 (ax)
X-Accept-Language: en-us, en
MIME-Version: 1.0
To: Mike Spille 
Subject: Re: Any comment?
References: <6.0.3.0.2.20040518011034.02008008@mail.krisnmike.com> <40AA0A64.6030300@jboss.org>
<40AA11D8.6050106@jboss.org> <6.1.0.6.2.20040518102643.026903e8@mail.krisnmike.com>
In-Reply-To: <6.1.0.6.2.20040518102643.026903e8@mail.krisnmike.com>
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
X-OriginalArrivalTime: 18 May 2004 16:30:18.0289 (UTC) FILETIME=[6802E210:01C43CF5]



Mike Spille wrote:

> Come on Bill, as they say in the movies, the gig is up.  A large number of people know that you did
> the Joe Murray posts.  The proof is more than conclusive enough.  Either fess up or don't, but don't
> try to walk this fine line that you're attempting here.  It is clear as day that you are Joe Murray.
> The only question is whether you will admit to it or not.


Honestly, I would be much more forthcoming about my opinions and knowledge of this "gig", but I do not
trust you or anyone else in your blog "crowd" one bit.   I believe that you would distort the true intentions
of anything I wrote and twist it to further your own personal agendas and/or vendattas.

> It looks 10 times worse, of course, when you couple it with other aliases by other JBoss people.  Ben 
> has been conclusively caught masquerading as Arun.  Everybody knows who Chip Tyler - and others - are.
> In the most positive interpretation, your actions as Joe Murray are the best of the worst.  The least
> objectionable actions out of a whole string of objectionable actions.
>
>With that said, several people I've talked to actually believe you've been more ensnared by the JBoss
> distortion field than anything else.  A direct quote from one person on this, talking about you:>
>
> "At the risk of sounding like Luke Skywalker, 'I think there is still good in Bill'.
> He realizes he got caught in the dirty little game but I'll give him the benefit of the doubt for
> being in the distortion aura of Marc.  That being said, the greatest proof of maturity is coming clean
> and apologizes by recognizing he used a fake alias".


I really could care less that others think I'm this poor naive soul ensnared in the distortion aura of Marc Fleury.

> No, this is not me making this up - someone else said this about you.  I just happen to agree.
> To a large extent you've obviously tried to do the right thing, but you are getting sucked down by
> the company you keep.  If you came clean about this in public it would raise people's opinions of
> you tremendously, believe it or not.  Many people understand technical disagreements - they do not
> understand, or condone, things like fake anonymous posts.  This issue isn't going to go away, it's
> only going to get bigger over time, and how you deal with it is going to define how the community at
> large thinks of you.

Go ahead, continue your crusade if it makes you feel important.

Bill
It was after these e-mails that I got Ben's apology e-mail. I forwarded Ben's e-mail to Andrew C. Oliver and Bill Burke, with this preface attached to Ben's e-mail:
To: acoliver@jboss.org,Bill Burke 
Subject: Fwd: Embarrassed and truly sorry


In case you guys weren't in the loop for this.  Incidentally, Ben as Arun was posting for over 6
months from both home and the JBoss headquarters.  I find it a little hard to believe that in all that
time no one else at your corporate headquarters had any knowledge of this.  And Bill, I concur that
the posts from Joe Murray were never close to the level of Arun's, not even close - but they'll be
tied together nonethess because "it's all JBoss".

        -Mike
The e-mail apparently bounced in the send to Oliver; I know linuxintegrators.com, his website, was down for some reason, so that may be why.

But awhile later I received this acid-etched response from Bill Burke:

On Rickard: [[Deleted MWS]] On Cameron: Did you know that JBoss Cache competes directly with (in)Coherence and we even have a customer that switched from it to JBoss Cache because of cost? On CDN: Did you know Marc lost his shirt in JBoss's first attempt at a company back in 1999 when he employed the likes of Rickard? So much so he had to move in with his in-laws in Atlanta? Yes, I know, cry-me-a-river, but that should put in perspective Marc's overall contribution to our company. So, on to CDN: Dain Sunstrom: [[Deleted MWS]] Jeremy Boynes: [[Deleted MWS]] Greg Wilkins: [[Deleted MWS]] Hiram Chirano: [[Deleted MWS]] Jason Dillon: [[Deleted MWS]] On ELBA: Did you know that elba was the island Napoleon was banished to? On Geronimo: Did you know that we obtained the original source submission of Geronimo code to Apache and that it had references to classes prefixed with JBoss? Did you know that code in Geronimo was within JBoss CVS with clear comments from Dain that it was a derivative of code that I and others had written? Did you know that the original Geronimo submission was even more similar in structure and design to JBoss? Did you know that our lawyers (who were actually lawyers who worked on FSF license cases and defended MySQL in their suit against Progress) told us that we should only focus on copyright infringements in that derivative work was much harder to prove? On revoking CVS rights: Did you know that we did not revoke any CVS rights until we found out about Geronimo and Elba? Once month after the split? Anyways, have fun. I'm sure you'll have fun forwarding this to your buddies. Bill Mike Spille wrote: In case you guys weren't in the loop for this. Incidentally, Ben as Arun was posting for over 6 months from both home and the JBoss headquarters. I find it a little hard to believe that in all that time no one else at your corporate headquarters had any knowledge of this. And Bill, I concur that the posts from Joe Murray were never close to the level of Arun's, not even close - but they'll be tied together nonethess because "it's all JBoss". -Mike Return-Path: Delivered-To: mike@krisnmik.user Received: (qmail 7168 invoked from network); 21 May 2004 19:30:51 -0000 Received: from unknown (HELO iss05.interliant.com) (198.3.182.195) by server256.com with SMTP; 21 May 2004 19:30:51 -0000 Received: from EX-008.mail.navisite.com (ex-008.interliant.com [198.3.179.39]) by iss05.interliant.com (8.10.2/8.10.2) with ESMTP id i4LJU5S29272 for ; Fri, 21 May 2004 14:30:05 -0500 (CDT) Received: from jboss.org ([24.128.152.56]) by EX-008.mail.navisite.com with Microsoft SMTPSVC(6.0.3790.0); Fri, 21 May 2004 14:27:48 -0500 Message-ID: <40AE58B1.9080605@jboss.org> Date: Fri, 21 May 2004 15:29:53 -0400 From: Bill Burke User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.4) Gecko/20030624 Netscape/7.1 (ax) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Mike Spille Subject: Re: Fwd: Embarrassed and truly sorry References: <6.0.3.0.2.20040521133540.02bef4b8@mail.krisnmike.com> In-Reply-To: <6.0.3.0.2.20040521133540.02bef4b8@mail.krisnmike.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 21 May 2004 19:27:48.0136 (UTC) FILETIME=[B30DD280:01C43F69] Mike, PLEASE read this carefully, as I have chosen each word very carefully. I long ago gave up on giving you the benefit of the doubt. I see you as somebody whose sole motive is to destroy what we have worked very hard to build and to build a name for himself in the process. Basically, I see you as a brownshirt. You have continually twisted the truth to your own purpose and continually FUDed us on many many issues. Some examples as saying: we attack anybody who answers questions on our forums, that we would do a Lutris on JBoss, that there was no development going on in JBoss, that Hibernate/JGroups would become dependent on JBoss, etc. etc. etc. I could go on forever..... You may argue with me on that point, but that is the way I see it, and every single person at JBoss Inc. see it, and the way other JBoss supporters see it. This is why Arun/Ben "attacked" you the way he did and questioned your existence. I had done the same in the past AS MYSELF, because, we just couldn't understand how somebody we had never interacted with could come out of the blue with such vice towards us. But I eventually came to the conclusion that you were brown shirting, trashing us to make a name for yourself. I believed this even more when I saw that you quit your job. I HOPE you have read the above very carefully as it is what I believe about you. As far as aliasing...As you know, I have never been afraid to call you out on any bullshit you spill out on TSS or even in your BLOG. What you need to understand is that you cannot post as yourself all the time because of who you are or because of who you work for. You, being an unemployed schmuck have nothing to lose posting as yourself. Personally, I have a LOT of anger built up and it has taken a lot of control to quell the things I know. Some of it seaped out in posts on TSS, some of it in private conversations with Dion or Cameron, some of it while teaching JBoss trainings. AND NOW I WILL SPILLE all of it here: On JBoss AOP's inactivity: [[Deleted MWS]] So you can take your CVS watching and SHOVE IT UP YOUR ASS!
As you can see, most of the latter half of the e-mails has been deleted by me. If you saw the rantings and ravings in them, you'd understand why. As for his claims about Geronimo - these have been rebutted in full by the ASF quite awhile ago, and I note that JBoss Inc. or any JBoss committers have not made any further claims.

Oh, and for those not aware of it, calling someone a "brownshirt" or accusing them of "brownshirting" is a reference to Nazi SS soldiers who were distinctive for their brown garb. So, apparently, I'm a Nazi. I had never heard of the term until someone else pointed it out to me today.

Getting back to Ben for a moment, here was my response to his apology:

To: "Ben  Sabrin" 
Subject: Re: Embarrassed and truly sorry


Ben, thanks for sending this.  I can imagine how hard something like this must have been to
write, and especially to hit the "send" button.  Let me say that I appreciate you acknowledging
what happened.

I understand in this situation you have to be careful about what you say, but some of this does
not add up for me.  It's very gallant to fall on your sword for the company, and most people wouldn't
be able to even go that far.  But I find it rather hard to believe that it was done without anyone else's
knowledge, or at least tacit consent.  The "Arun" posts ran from August 2003 to March 2004, 7 months, and
from both your home and JBoss HQ.  And those posts were in the same threads as "Chip Tyler", Bill Burke as
himself, and Bill Burke as Joe Murray.  It seems to be me there had to be some knowledge in the company
about this.  This is more than a Saturday mistake.

For the record I don't believe you could possibly be tied to all of the personalities that all
originated out of JBoss HQ, or from other homes than yours, and find it a tough sell that so many
personalities were all created independently without anyone at JBoss being aware of the others.  

And of course none of it jibes with the PR response on the JBoss blog, but that's to be expected.
:-)  On that note I think your company made a huge mistake by not just admitting it and saying it
wouldn't happen again.   The PR speak in the blog entry is a disaster in the community.

I sincerely appreciate that you're acknowledging what's happened, but I fail to see why you'd protect
others that have done the same thing, or the company that allowed it to happen.

On the CDN stuff - I've talked to some of your people and some of the CDN people as well, and the
best I can say is that the two sides tell radically different stories.  I wasn't there, so I can't
tell who's "right".  That there's bad blood, there's no doubt.

        -Mike
This was Ben's response:
Return-Path: 
Delivered-To: mike@krisnmik.user
Received: (qmail 14296 invoked from network); 21 May 2004 19:19:04 -0000
Received: from unknown (HELO iss05.interliant.com) (198.3.182.195)
  by server256.com with SMTP; 21 May 2004 19:19:04 -0000
Received: from EX-007.mail.navisite.com (ex-007.interliant.com [198.3.179.37])
        by iss05.interliant.com (8.10.2/8.10.2) with ESMTP id i4LJIGS19317
        for ; Fri, 21 May 2004 14:18:16 -0500 (CDT)
Received: from EX-001.mail.navisite.com ([172.16.1.11]) by EX-007.mail.navisite.com with Microsoft SMTPSVC(6.0.3790.0);
         Fri, 21 May 2004 14:15:56 -0500
X-MimeOLE: Produced By Microsoft Exchange V6.5.6944.0
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/related;
        type="multipart/alternative";
        boundary="----_=_NextPart_001_01C43F67.EAF39DD4"
Subject: RE: Embarrassed and truly sorry
Date: Fri, 21 May 2004 14:17:05 -0500
Message-ID: <75653EEEF6218646B5239EA1D5204E2602837E55@EX-001.mail.navisite.com>
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator: 
Thread-Topic: Embarrassed and truly sorry
Thread-Index: AcQ/Y8DTwWcliPsfRrmpDbRtfByg3AABAU/g
From: "Ben  Sabrin" 
To: "Mike Spille" 
X-OriginalArrivalTime: 21 May 2004 19:15:56.0235 (UTC) FILETIME=[0ABA59B0:01C43F68]

Mike, 
 
Believe what you want, but what I do is my business not the business of JBoss.
This is not a plot, nobody but me even knew of Arun.  And Mike on the CDN stuff, I know
what I know, Dain resigned through a press release.  That is the most unprofessional thing
in the world.  
 
And no, this email was not hard to write, as I am not someone who is afraid to admit his
mistakes, but hitting the send button knowing you will most likely continue to try and use
this against us was the real hard part.  Hopefully, you can prove to me that we have both
started to move past this and these emails stay between us. 
 
Regards,
 
Ben Sabrin
Director, Sales North America
404-467-8555 x202
404-664-9466 mobile
404-948-1496 fax
ben@jboss.com
It was only slightly later when I forwarded Bill Burke's acid-etched response to Ben Sabrin. The purpose of this was simple - Bill Burke sure seemed to know about the "Ben/Arun" link, so why was Ben saying that "nobody but me even knew of Arun"? Well, after I showed Ben Sabrin the e-mail I got from Bill Burke, this was his response:

Return-Path: 
Delivered-To: mike@krisnmik.user
Received: (qmail 27308 invoked from network); 21 May 2004 20:54:14 -0000
Received: from unknown (HELO iss05.interliant.com) (198.3.182.195)
  by server256.com with SMTP; 21 May 2004 20:54:14 -0000
Received: from EX-008.mail.navisite.com (ex-008.interliant.com [198.3.179.39])
        by iss05.interliant.com (8.10.2/8.10.2) with ESMTP id i4LKrSS18091
        for ; Fri, 21 May 2004 15:53:29 -0500 (CDT)
Received: from EX-001.mail.navisite.com ([172.16.1.11]) by EX-008.mail.navisite.com with Microsoft SMTPSVC(6.0.3790.0);
         Fri, 21 May 2004 15:51:00 -0500
X-MimeOLE: Produced By Microsoft Exchange V6.5.6944.0
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
        charset="us-ascii"
Subject: RE: Re: Fwd: Embarrassed and truly sorry
Date: Fri, 21 May 2004 15:52:28 -0500
Message-ID: <75653EEEF6218646B5239EA1D5204E26028380D0@EX-001.mail.navisite.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Re: Fwd: Embarrassed and truly sorry
Thread-Index: AcQ/dOcPkR5Ycul0RgmErU5ApU9/MgAAJLwg
From: "Ben  Sabrin" 
To: "Mike Spille" 
X-OriginalArrivalTime: 21 May 2004 20:51:00.0731 (UTC) FILETIME=[52DF5CB0:01C43F75]

Bill knew because I told him, but not because this was a JBoss policy...

Ben Sabrin
Director, Sales North America
404-467-8555 x202
404-664-9466 mobile
404-948-1496 fax
ben@jboss.com
 


-----Original Message-----
From: Mike Spille [mailto:mike@krisnmike.com] 
Sent: Friday, May 21, 2004 4:46 PM
To: Ben Sabrin
Subject: Fwd: Re: Fwd: Embarrassed and truly sorry


Ben, I forwarded your post to Bill Burke.  This was his (rather
vitriolic 
reply).  It's hard to read through all of Bill's anger, but he seems to
be 
admitting that he knew about the "Arun" alias.

         -Mike
So according to Ben, Bill Burke knew about the Arun Patel alias. That's two senior executives at JBoss who knowingly undertook these practices, and one knew about the other. It's not a huge stretch to imagine that Ben Sabrin knew about Bill Burke's alias as well.

This begs the question - how many senior executives does it take, all doing the same thing, before you have a de facto policy?

So there you have it sports fans, in all its ugly glory. Ben Sabrin has outright admitted that he posted under fake aliases on TheServerSide.com, and further admitted that he did so to try to erode my professional credibility. Bill Burke has done a remarkable dance of tip toeing around the core issue; in my read, he's effectively admitting to having posted under fake aliases, but he never (quite) comes out and says it. I'll leave it up to the reader to decide. Clearly Bill Burke knew that Ben Sabrin was posting as Arun Patel, and apparently saw nothing wrong with it. I'm grateful to Ben that he had the courage to come forward and admit to the truth, and to also apologize for his actions. The same cannot be said for the rest of the JBoss crew.

These of course represent only two of the aliases originating from JBoss Inc. servers and/or home locations of JBoss senior management. On top of these admissions, I have no word on aliases like Chip Tyler or other aliases that have been exposed, but they are real and they do exist. I also have zero reason to believe that aliases like Chip Tyler belonged to either Ben Sabrin or Bill Burke - all the evidence from on-line logs that I have seen indicate that they are in fact tied to Marc Fleury. I can't offer direct evidence to that fact, but I look to others (and you know who you are!) to step forward and show what proofs they have on this matter. I do know that Mr. Fleury avoided any kind of direct admission in his "response" on these allegations - feel free to read his "Response" for yourself. But now, it's crystal clear that one of Fleury's top management people was involved in smearing people's reputations using fake names. I believe from reading Bill Burke's e-mails that he's also guilty of also posting under a fake alias; but to his credit the content under that alias was much more mild and not directed at destroying JBoss critics' credibility. That was "merely" astroturfing. It's plain that Bill Burke knew of Ben's fake alias and what Ben was posting. I make the assumption that if Bill knew, others knew as well (JBoss Inc, after all, is not all that large of a company). In fact I would say that it stretches the bounds of reasonability to the breaking point to assume that other people in senior management at JBoss did not know what was going on.

This ultimately begs the question: what did the CEO, Marc Fleury, know, and when did he know it? And, I have to ask, what's he going to do about it? Yes, I saw the "response" - and it's as close to a non-response as you could possibly get short of just banging our random words.

Yes, I realize that some people will look at this article and claim that I have broken trust somehow, that private e-mails should be inviolate, you shouldn't post without the author's permission, etc. My viewpoint on this is simple: employees of JBoss Inc. broke my trust a very, very long time ago, and continued to do so with their attacks from fake aliases on TheServerSide.com.
Posted by Mike Spille in General at 20040524 Comments[36]
20040518 Tuesday May 18, 2004

Hippie Update Dale Asberry has responded to my own and others criticisms in a new blog entry on Artima. I'll deal solely with his responses to my blog entry and a post I made on the Artima forums.

Update
It's worthwhile to correlate Dale's words with his resume, you can find it on his website. This isn't a judgement on him, but an observation that ties into his ideas on rewards: Dale has averaged 1 job per year for the past 6 years. Several of those are sub-1-year gigs. Sorry, but he hasn't stuck around long enough anywhere for any sort of rewards mechanism to even kick in.

Dale starts out by defining coercion vs. manipulation - complete with the dictionary definitions :-) He sums it up by saying "The important distinction being the use of force versus the use of subtle, crafty techniques to get what you want".

What Dale is missing here is that bonuses are not a manipulative tactic. As I said in my Artima forum entry, a bonus is a form of prid quo pro - do this for me and you will get that. Let me state this clearly: this is not manipulation. An agreement between adults is not manipulation, it's an agreement.

Dale goes on to say:

Study after study after study have conclusively shown that doing nothing is more effective than "paying for performance". I would strongly recommend that you read my original reference - it will point you to the quantitative evidence.
As I mentioned in my entry, I consider the studies Dale refers to as wholly unscientific. I will go beyond this and be blunt: these studies are mostly done by people who make $40K-$60K a year. This is the entry level salary of a software developer. I know people who's bonuses are higher than these people's salaries. When these experts are in my environment and making the same money as me, perhaps then we can talk. This is exactly equivalent to an analyst at Gartner pulling down $75K a year telling a multi-billion dollar company how it should conduct its business. If the Gartner analyst knew anything about the business, he'd be making a helluva lot more money than he does.

There's a brief interlude about children. I think this is largely irrelevant - I'm a 36 year old man, not an 8 year old boy who hasn't learn not to stay up past his bedtime. Nonetheless, he says "Let me clarify then: children should neither be punished nor rewarded". I have met these sorts of children. There is a common word for describing them. That word is "brat". They are disconnected from the world around them and believe that the world owes them a living. This is the child that will grow up to sue a knife manufacturer for making the sharp knives that cut their hand off because they were being stupid.

On the subject of firing, Dale says:
To be clear, I have no issues with firing someone. On the other hand, (strangely/amazingly) I've never had to - simply suggesting that the individual is not a good match for the project, team, company, etc has been enough to get them to leave on their own volition. I got them to leave and they got to do it on their own terms. Did they learn a lesson? I hope so, but ultimately, that's up to them.
Hah! Every place I have ever worked has had the hangers-on. These are dull people who do mediocre jobs who do not care about raises, bonuses, or what people think of them. Having a manager take them aside "and suggesting that the individual is not a good match for the project, team, company, etc" has absolutely no effect on them.

Dale seems to believe in positive communal psychology. Well, if I speak to a person and tell them they're not cutting it, of course they'll quit and go find a better job. Yeah, sure. This completely ignores reality - most people work to put a roof over their heads, and food on their tables, and to take a nice vacation twice a year. Wake up Dale - people do not work out of good will and a need to succeed. They work so they can buy things.

On the same subject, Dale goes on to say:
Regardless, "firing" and "bonus" are not physical feedback beyond being tossed out the door or being handed a check. They are both psychological power-plays that make the person doling the punishment/reward feel like they are in control.
This is a load of horseshit. Being fired is not a "psychological power-play". Earth to Dale: if you get fired, you must find a job or you will starve. This is not psychology, this is life. If someone gives me a $50K bonus, I may have enough money to buy a house. This is not psychology - this is real life. Further, as a I said, a bonus structure is an agremeent between adults. If I work on Wall Street, I knowingly take a lower salary with an agreement about how bonuses will factor in. The agreement is that if the firm does well and I do my job well, I may get that $50K bonus and be able to buy that house. This is not a control issue, this is not psychology - this is something that Dale may have heard of called "business".

With that behind us, we move on. Dale says this:
I've never suggested, even remotely, that feedback should be removed. My managerial success has come from building trust and respect with my reports. I evaluate their abilities and assign them tasks. We negotiate deadlines and I hold them responsible.

Over time, I've realized that most feedback is from them to me: technical problems they encounter, frustrations with coworkers, and interference caused by business processes. My job (as a manager) is to provide direction, eliminate roadblocks, and just listen.
The above quote is a fundamental hole in Dale's argument. Can you spot the hole? Here I'll give you a minute to re-read the above and think about it (la la la la la la la, [keep reading/thinking] la la la la la....).

OK, have you spotted the hole? Well, here it is: Dale says "We negotiate deadlines and I hold them responsible". Here's the million dollar question: how do you "hold them responsible" in a system with no punishment and no rewards? How does an individual have responsibility when they take no risk? How do you define responsibility when there can be no consequences?

I'll render an example: you have to go somewhere, and you ask someone to watch your dog for the day. "I hold you responsible for this dog, Joe". Let me render two scenarios here.

First, you come back and your dog is gone. What do you do? You told Joe he's "responsible", but according to Dale we can't punish him. So what does "responsible" mean? In this context, responsible means you trust the other person to do something for you - and they must do it with no expectation of reward or punishment.

OK, so second. You come back and not only is your dog still there, but he's been washed, fed a good meal, and is obviously abundantly happy. What do you do? Joe has done an outstanding job of being responsible. According to Dale, the best you can do is say "Thanks". You can't offer to do something for Joe, or give him money - no, that's a "reward".

In these scenarios, Joe has no motivation beyond the goodness of his heart. If he loses your dog, it's no skin off his nose. If he treats your dog really well, likewise nothing happens. Tell me Dale: you think Joe is going to continue treating your dog well if you keep asking him to watch him for you? You think he will continue to go the extra mile - that he craves you saying "thanks" so much that he'll do that? Might not Joe get resentful and not treat the dog so well - or refuse?

The plain fact is, is that few people will do something for nothing, and that is precisely what Dale is saying the best working environment is. Dale tells us that people will do their best when we make it crystal clear that we will not reward them for their efforts.

After this there's alot of references to the (again) unscientific and biased studies. There's one gem in there, however. Dale says (and also quotes others):
As I stated above, "the fact is that an organisation employs individuals and then puts them into a system where they can perform no better than the system allows. The mistake then is to assume that any problems with the system are the fault of the individual - in other words, the system is okay but you're not doing your job right."
Dale, what you're quoting here is a fundamental phenomena of late 20th century American culture onward: the individual has no responsibility. No one is at fault. You are explicitly saying that you can't fault an individual for failing, but that it must be the system that failed the individual. This argument has been used to excuse every piece of irresponsible and wreckless behavior known to man. This is a fancy, PhD way of playing the blame game: don't try to fix a problem, instead find someone to the blame. Let me re-ask the question I posed before in this slightly different context: how can one have responsibility if you can always lay the blame on the system?

This can also be restated in another startling way: corporations exist as a collection of individuals. Think about this for a moment, Dale: "The System" you are talking about is nothing more than a collection of individuals. You may have a hard time seeing this because of the way you're wording your arguments. When you say "The System", you're making it inhuman. You are implying that employees have no power. The reality is that "The System" is made up of a bunch of people. And an employee can negotiate (or renegotiate) or walk. They do have power. Just as the employer does (which again is a person, not a Faceless System). When you speak of manipulation, again you assume the employee has no power. Get it straight: an agreement is not manipulation.

I'm not going to go point-for-point on the rest of the article for the simple reason that all of the remainder of the article is based on Dale referring to blatantly unscientific studies made by people who make very little money, and have never experienced a bonus system.

Again and again, Dale simply ignores the lessons of capitalism: people work best when they see a tangible return on their efforts. Emphasis on tangible. Worse, Dale's "studies" still completely ignore human nature - many people just will not go the extra mile without a quid-pro-quo. In fact, people will take advantage of the ideal systems that Dale describes to get something for nothing (or, more reasonably, something for very little). The Utopia Inc that Dale describes is a perfect haven for the lazy and incompetent.

I'll leave you with this: there's a big kernel of truth in what Dale says, but he makes his mistake in taking an absolutist position. What most of Dale's "studies" show is that tangible rewards alone are not enough. This is most definitely true. But Dale refuses to see the flip side of this: The lovely intangibles that he describes are also not enough. The best companies are the ones that balance both: they encourage esprit de corps and feeling good about your job, and they reward people for doing their jobs well, and they punish people for failing at their jobs. One side or another is not enough - to make people happy and productive you need all the pieces of the puzzle. I'll also say this: people who argue against bonuses are universally people who have never gotten a significant bonus. I put this to Dale: If you worked for a company, did an outstanding job, and was offered a $50K bonus - would you turn it down? Would you feel bad about getting that $50K if you felt you really earned it? Is the problem here really rewards, or is it that rewards haven't been significant enough to matter?

Posted by Mike Spille in General at 20040518 Comments[10]
20040517 Monday May 17, 2004

Anonymous Fakers: Update! TheServerSide.com has just released a new feature that allows users to see not only what threads an individual has posted on, but also what other users have posted from the same location.

As it says in the announcement on the serverside:

The "User Threads" view shows the various threads that a community member has participated on. We have added some more information below the threads, which shows a list of other community members who have posted from the same location as this user. Users could be linked due to a variety of reasons, including posts made from a shared IP (dialup, dynamic DSL/broadband, wireless location), multiple aliases per person, etc.

The community has let us know that they don't like "fake" accounts posting on behalf of certain parties. We know it happens a lot by a lot of groups, and we know that we can't totally stop it, however this will hopefully help a little.

The linkage may NOT be due to fake accounts (as mentioned above), so please do not jump to any conclusions.
Given how this works, many people will legimately show up using the same IP address (most often because they share a firewall). For example, my entry shows: "Kenny Dao, dharmesh sethi, Glen Klyuzner, Syed Taqvi". These means that I shared an IP with these people. I happen to know a few of these people, they worked for the company I used to work for.

But likewise, you can do checks on other people. You can run a check on someone like, say, Bill Burke, and see who was posting from the same IP as he was.

This is one piece of evidence you can use to help decide who's who, or if nothing else who have shared a common location at some point. No doubt TSS keeps detailed logs of who's posting from where, so they can track this sort of thing at a finer granularity (and likely have at some point). Other information may also be used to determine who's who, and to seperate the wheat from the chaffe.
Posted by Mike Spille in General at 20040517 Comments[2]

The Anonymous Fakers For well over a year now, a little battle has been waged on TheServerSide.com. In the grand scheme of things, it's a tempest in a tea pot, a parenethetical comment in a footnote to an addendum in the histories of software development. But to those caught in the middle, it's real enough, and big enough to become a serious issue and one that deserves some attention.

For my own part, it started simply enough. I had gotten involved in the serverside because I liked the content, liked what some people were saying there, and it contained a goodly variety of topics which were discussed and debated. For me, it was on-point, and was kept fresh and alive by the editors there and the community which they interacted with.

Over this span of time, from then to now, it was mostly an enjoyable experience, both in my own posting on TSS, and seeing what others had to say on the same topics. A key part of learning is not just doing, but also interacting with others. But there were some troubling and puzzling aspects to TSS interactions. No, I'm not talking about arguments and disagreements and flame wars. These aspects of on-line discussion are a natural consequence of the human condition. I'm sure people were arguing and bickering at the very beginning of time, and will continue doing so right up to the heat death of the universe. They can be intense, but they're real and a fundamental part of who we are. So I may have been passionate and aggressive in arguing with Jeurgen Hoeller, or Rod Johnson, or even Cameron Purdy or Bela Ban, but never puzzled or troubled. They were arguments, even flamewars of an almost grandiose scale, but they were real.

The puzzling part, and the troubling part, were the discussions which were not real. Sounds weird, doesn't it? How can you have not-real discussions? Well it's quite simple really - such unreal discussions arise naturally when one or more of the participants is masked, and are pushing an agenda that is hidden from the world to see. When these masked marauders enter a discussion, you are no longer debating facts and opinions; instead, you are fencing with a phantom; you are hamstrung and blindfolded, and are reduced to spending much of your time peering through a murky fog and trying to decide what is real, and what is not. You are no longer debating facts, or trading honest opinions; instead, you are fighting against an agenda which you cannot see, against an unknown number of false names and personalities who might inject themselves into a discussion at any point. You are no longer arguing anything real; instead you are mostly arguing against fictional characters with fictional stories. You see, the Anonymous Fakers change everything in on-line discussions.

Starting from the End, Working Towards the Beginnings

Up to now, I've been speaking in the abstract; now it's time to get specific and concrete. The paragraphs above are the setup, now it's time to get to the punchline. To make things as clear as possible, I'm going to take a somewhat unorthodox approach and begin at the end, and then unravel backwards in time.

In the recent TSS discussion IDC Reports on J2EE application server market Cameron Purdy dropped a little aside at some of the comments written by one "Randy Spears". Quoth the Purdy:
You were doing OK until you said "better servers" and "JBoss". I think that one blew your cover.
This comment drew the ire of Corby Page, who responded:
A quick scan of previous threads indicates that you have previously challenged Mr. Spear's identity.

There seems to be a bit of a pattern that if someone posts something you don't agree with, you accuse them of not being a real person.

Do you understand why this is a bit offensive?
To the innocent reader, it would seem that Mr. Page has the right of it - it just doesn't do to go randomly accusing people of being anonymous shills for some company. But Corby wasn't aware of the deeper background that was going on beyond his sight (indeed - this is a fundamental piece of wrongness that infects disscusions involving Anonymous Fakers - so much is hidden from the casual reader), and Cameron clarified this in a follow-up:
If a corporate entity makes it an endorsed policy to post under fake names on a public board then this is what they should expect. When that corporation's employees, on that corporation's equipment, operating from that corporation's offices anonymously attack -- on a public forum and using pseudonyms -- people that they disagree with, as well as competing products and companies, then I feel quite justified in disbelieving every blindly positive post made by recently created accounts seemingly named after porn stars. Do you understand why this is a bit offensive?

If you don't find those posts offensive, then you're sick, because when a community (which includes you) tolerates such twisted contributions, it cannot but weaken the very fabric of that community. If you don't like it, fix the real problem. Don't get high-and-mighty pointing your finger at people who stand against such blatant dishonesty.

I will personally put my real name on this message, and stand behind it.
The above apparently isn't enough to convince Corby, and Corby challenges Cameron effectively to put up or shut up. Cameron chooses the former:
Fair enough. In this case, the fact that Ben signed up Arun Patel's account using his real email address was a pretty good indicator of who he was. After he went after Mike Spille (for pointing out problems with JBoss,) a little digging was done, and *voila* that's what was found.

It was already known for some time that posting under false names was "official policy." That information (as well as some more interesting factoids) came out when CDN split from JBG.

The number of complaints going to TSS about these various obviously faked posts were impossible to ignore, so they set out to fix the problem. Unfortunately, that's about as much of the story as I can tell you myself. The rest will have to come from them, as I believe they plan to announce their findings at some point in the near future.

As for the merits of posting anonymously .. well, I have posted anonymously on the internet before, so I'm not suggesting that there isn't a time and place for anonymity. However, this pattern of repeatedly attacking individuals and competing products from behind the veil of anonymity is far beyond the domain of fair play and ethical behavior. And trust me, it's not one person doing it, lest some trite official apology try to pin this tail onto a single donkey. Randy Spears, for example, was posting on one of his five accounts, and he wasn't posting from the JBoss HQ (like Arun Patel -- supposedly in India -- does.)

Corby, I want you to go back and read Arun's personal attacks on Mike and tell me what you think. Then, I want you to consider that Mike knew that Arun was a JBoss employee all along. Tell me, which was offensive, and who was the better man?

Getting Down to Brass Tacks

That last post from Cameron puts most of the cards out onto the table. Here's a quick summary of the allegations:
High-level employees of JBoss Inc (formerly JBoss Group) have been posting under fake names on TSS. These anonymous fakers have posted two types of posts: posts intended to create the illusion of many people who have used JBoss and like it (positive posts promoting their agenda), and posts intended to undermine competitors and naysayers (such as yours truly) - the negative muckraking posts. Such posts are not confined to one individual. There is no lone whacko in the company doing this. In fact this appears to be a fundamental policy of JBoss Inc - use fake posting personalities to further the JBoss Inc. agenda and to spread FUD about JBoss competitors and detractors. In fact, some of the fakers are in the highest echelons of the JBoss Inc. organization. This is a coordinated, deliberate policy. In some cases, evidence points directly at specific JBoss Inc employees. The best and clearest example is the link between Arun Patel and Ben Sabrin - whose title is Director of Sales and Business Development. Other examples, such as the link between a certain JBoss employee and identities such as "Joe Murray", are also strong but not as iron clad as the Arun Patel example. And finally, there are a number of identities that can't be tied to an individual, but clearly originate from common JBoss servers.
You can read some of the material that posters like "Joe" and "Arun" have submitted by going to the userthreads page of TSS for them. Here's Joe's stuff and Arun's stuff. These are, of course, the tip of the iceberg. There are other fake names involved as well. Some of them clearly are fake names, and show that the individual purposefully want to be anonymous. Others are plain old names that seem as real as yours and mine.

Once I learned that there was hard-core proof of these fakers and their links to JBoss, a whole lot of that puzzling and troubling about past TSS discussions clicked for me. Many of those discussions where JBoss was getting praised and people like myself or CDN members were being dissed - so much of that wasn't real, it was a deliberate web of lies, deception, and FUD driven by individuals motivated by corporate greed and ego. So much of my time, and others, wasted fencing with phantoms.

The net result of all of these fakers is that a false impression is given of JBoss being a pervasive, widely used product which its users love, and a huge amount of FUD is spread about JBoss competitors and detractors. Some of it clearly was "just business" - just going out and promoting their product under false names and spreading light weight FUD about competitors. But some of it was personal, nasty, and malicious. Arun nee Ben Sabrin went so far as to claim he had called my company and NYC telephone companies and found definitely that I did not work for who I said I did, and that I probably didn't even live in NYC! Here's a director in the JBoss organization, a complete faker claiming that he's "Arun Patel", an Indian working for WIPRO in Bangalore, India - accusing a real person of being a faker! Here's some of Arun's posts on this topic:
This guy is unbelievable, I tell you. I do not think I have ever met anyone quite as bull headed as this guy. Even if he is dead wrong, he believes he is right. Perhaps, Mr. Spille should be in the legal profession instead of living his glorious life in a cubicle. However, he does have a lot of hate deep down hate for JBoss. So, I am in the process of doing a little experiement. First off, I am going to call his so-called employer and see if there really is a Mike Spille there. If not, I have a strage suspicion that he could be one of the defectors of JBoss just can't grasp with the fact that the project has continued after their fork. My hypothesis could be wrong, but curiousity is killing me. I have a hard time seeing people so hateful in the world.

[...]

There is no Mike Spille at SIAC, which is the company he said he worked for in a previous post a while back. He said he worked for the company that developed the technology for the NY Stock Exchange. There is a chance that he is not a subscriber to this system. Now the second test, calling information for Spille in NYC...

[...]

Called information in New York city for Mike Spille and there is no listing. Once again, it is possible he is not listed, but even if he is not listed I would have received a recording that says, at the customers request, this number is not listed. But they could not find a Mike Spille within 30 minutes of NYC. According to his blog, that is where he lives and works.

Now I am really starting to get suspicious.

[...]

Yes, sir I have seen your resume and website, but I still do not buy who are who you say you are. Who are these people that can vouch for you. I have run a search for technovision and found nothing....
Ironically, the thread in which this occured (JBoss closes $10 million in venture financing) is also one where "Joe Murray" put in his two cents as well. Some of Joe Murray's comments:
Yeah, no one has also mentioned that JBoss is profitable right now. At least, that is what the press announcement says. Having worked for a few startups in the past, getting up to what, 20 employees without VC fundings is quite a feat. The real test will be to see if the VC's force them to burn that money.

[...]

It seems from the press release that JBoss Inc. is more than just JBoss. Hibernate, Tomcat, JBoss Nukes. How can JBoss Inc. claim Hibernate and Tomcat
?
JMS performs quite well in production with the more recent versions of JBoss. I've even heard some rumblings of JMS clustering being available in later JBOss releases. Anybody have any more info on that?

As far as distributed tx and recovery and logging goes, yes, that is a weakpoint, but how often do apps require that kind of functionality. My guess the number is in the minority. I guess what I am saying is that the JBoss J2EE stack is quite complete minus distributed tx and recovery based from what I've seen
.
[...]

These are my impressions as well Corby. JMS and CMP performance improved dramatically with JBoss 3.2.2 and beyond. Seems they got their act together finally on these components over the past 6 months.Like I said. JMS has seen dramatic improvements over the past 6 months, JBoss 3.2.2 and beyond. This plan looks like it was written last July. Same could be said for the CMP implementation. Vast improvements over past 6 months. So, if JBoss employees stated that JMS and CMP sucked awhile ago, it seems that they have worked to address the situation.
Here we have "Joe Murray" and "Arun Patel" interacting in this thread to pump up the JBoss image, and to simultaneously attack JBoss naysayers (and the attacks aren't just on me - look at previous Arun posts where he visciously rips apart the CDN folks). The worst part is that people like Corby Page have been sucked into these discussions and saw posts by people like "Joe Murray" as evidence of JBoss doing well.

Professional Open Source?

JBoss' new tag line is "Professional Open Source". To me this is an unimaginably ironic moniker, given the practices the company routinely engages in. From what I've seen, this company is neither professional or open. It's more a bunch of adolescents thinking that since they've gotten away with lying and deception this long, they can just keep it up indefinitely - the rest of us are such suckers that we'll never catch on to the truth. And the bad seeds there may be taking down the reputations and livelihoods of the innocents that may be employed there.

Somewhat tangental the above, Andy O e-mailed me just the other day, asking the question:
Do you really have an irrational hatred for JBoss or is that for attention/show (like Hani)?
I don't mean to necessarily imply that this gentleman is innocent or guilty (I have no idea if he's involved in any of this trickery), but certainly confusion is writ large in his question. To paraphrase and summarize my reply to Andy: "I don't hate JBoss or the people in it, but I do deplore the practices JBoss regularly engages in. And while hatred isn't involved, you could say that disgust is. Why a number of obviously talented people would sink to these lows is beyond my comprehension".

In conclusion, this is just my side of the tale. Others may have other views, and I look forward to hearing what they may have to say in this matter. In particular I hope that people from TSS step forward and clarify these matters in public. I would also like to see a public comment on this from JBoss people as well. If this is all a huge misunderstanding I would very much like to listen to JBoss explain it. And if not, I would very much like to hear why they would act in this manner. I don't expect such a response either way, but one can always hope :-) I'd like to also say that I am not against anonimity. I believe it has value, and that attempts to eliminate it are worse than living with it as it is today. Anonymous Fakers are bad, but trying to prove identity in all cases is even worse. But despite that, living with anonimity does not mean that we need turn a blind eye to abuse and ferret it out when we can.
Posted by Mike Spille in General at 20040517 Comments[15]
20040513 Thursday May 13, 2004

Treating Professionals Like Hippies Dale Asberry has posted an article on his Artima blog entitled Why salary bonus and other incentives fail to meet their objectives. The thesis of the article can be summed up thusly:

Rewards are bad. You shouldn't reward people for exceptional work - an equitable salary is sufficient. In fact, rewards make people less productive. Dale specifically tells us to get rid of bonuses, and get rid of at-risk pay. Employee evaluations are bad. He quotes a Big Brain on this, saying "W. Edwards Deming ... has called the system by which merit is appraised and rewarded 'the most powerful inhibitor to quality and productivity in the Western world.'". As a corollary to this, punishments are also bad, as are threats of punishment. Treat employees with respect. Managers should observe, listen, and encourage lots of feedback. Value Collaboration (people working in groups, not alone), Content (a meaningful job), and Choice (give the employee the power to choose as much as possible).
The article starts out demolishing the concepts of bonuses and rewards, ties this into the evils of employee evaluations,and then goes onto say what managers should do instead. On the "should do" list are a number of actions that can be summed as positive interactions - let people choose when they can, interact with them a lot, encourage them to interact with each other, and do what it takes to break down barriers and to make their jobs meaningful.

That's how Dale paints the picture. My own take, after reading the article through several times, is somewhat different.
Dale confuses bad rewards with all rewards. He uses examples of bad reward systems, and leaps to the conclusion that since some companies have bad reward mechanisms, all reward mechanisms are bad. Dale doesn't come out and say it, but he's fallen for the biggest fallacy in the western world, popularized by misunderstandings of Democracy: that all people are created equal. All people are not equal - they may have equal basic rights, but that does not mean that you should treat your super-genius mad scientist developer who wrote 3/4s of the system the same as the shaky recent college grad who's still not sure what 'static' means. Dale tells us evaluations are bad - and then fails to give us an alternative mechanism. Do people in Dale's mythical universe ever get fired? Can they be fired? How do people in DaleWorld get a raise? Do they get raises, or are raises an irrelevant monetary concern that distracts from their job? Dale tells us punishment, or threat of punishment, are bad. People should be encouraged, interacted with, gently turned from a bad course - never punished. This assumes that all employees are (again) equal, and always act in each others best interest, and actually do their jobs. It is assumed that all work that must be done can be cast in a light to make it enjoyable for the worker. As a corollary, any work that is not enjoyable presumably should be jettissoned - it's not enjoyable, you can't make people enjoy it, and you're never, ever suppose to coerce or control people (or reward them) for working on something unenjoyable. So, to paraphrase, if a task can't be made fun something must be wrong with the task, and you should eliminate it.
What Dale's really talking about here is treating professional employees like Hippies. It's a shiny, happy, carefree world, and work should reflect that. We should assume all people have Good Will towards all other people, we all "just wanna get along". Money, captitalism, and similar concerns should be eliminated - they are distracting. And no one is better than anyone else - all are equally valued, should be equally loved, and no head should rise above the crowd.

What's going on here is that Dale is taking some ideas from the open source world, injecting a big dose of Stallmanisms, and effectively proposing a Star Trek universe - people should be encouraged to do what they like, it is assumed that people will naturally work together well, money has been eliminated, and in the end we will all be collectively enabled to bring our best talents to bear towards bettering humanity as a whole. We will rise above our animal ancestry, shed the chains of bondage and selfishness, and boldly go where no man has gone before!!

Unfortunately for Dale, he's simultaneously ignoring human psychology, fundamental aspects of behavior and sociology, and blatantly ignoring the basic lessons of capitalism from the past century. Fundamentally, most of the adult human population are children psychologically - they have petty wants and needs and emotions, and when all external factors are eliminated they will tend most often towards vegetating on the couch with the remote in one hand and a beer in the other. Most people in this world need both the carrot and the whip to get them motivated - they need something _tangible_ to shoot for, a tangible reward for their efforts; and they need the downside too, the reality that if they fail Bad Things will happen to them.

There is another word for this hippie-ish sort of movement; a political theory that says that all people are equal, all should be rewarded equally, no head should stand above another, and each and all should work toward the collective good. I won't invoke that word, since it seems to charge discussions out of all proportion, but you can guess what it is I'm sure :-) But still - there's a reason why this political theory has never even been attempted in reality, and the perversions that have been tried have fallen down flat on their face. People are different. Some are dumb as tree stumps, some are so smart you wonder how they keep all that brain power in a vessel as small as the human skull. Some people are naturally gregarious, others are naturally loners. Some jobs are fun and invigorating and exciting - and some are dull, dreary, even dangerous - but they must be done for society to advance. The world is full of Einsteins, those guys from Dumb and Dumber, and uncountable billions in between.

The big problem with Dale's ideas and theories is that he focuses on the positives, ignores the negatives, and assumes that people are fundamentally motivated by Good Will and a sense of Esprit de Corps. Maybe he just had a really sheltered childhood, but what he describes isn't the real world. Sometimes, you need a genius to work on some of your systems, and that genius might be a cantankerous, nasty old man. So you gotta deal with it. Sometimes the nicest person, someone you could see as your best friend and secret confidant, is a complete idiot at their job and utterly incompetent - and whether you like them or not, you should fire them (yes, any viable system needs a mechanism to fire people). People can and do sometimes work solely for the good feeling of making a difference, of doing something worthwhile - but they will not do this indefinitely. Eventually, they'll notice that esprit de corps isn't enough. Eventually, they'll become bitter that they are in reality the star of the group, but they get paid the same as lack-luster average people who accomplish little of merit in their careers. One day, they'll sit up and notice that the company is raking in giant profits and they never get anything more than an "equitable" salary. And every group gets infected at one time or another with a poisoned individual who does little real work, aggrandizes themselves at the expense of others, and generally actively works to better themselves at the expense of the organization. For these people, the shiny happy "make love not war" policies don't work. Sometimes unpleasantness will infect an organization, and the organization needs to respond with unpleasantness to make the situation right again.

Dale is right that some organizations are too hierarchical, too bureacratic, and too impersonal in their approach, and this is bad for both the people working there and ultimately for the organization. But Dale makes the classic error of knee-jerking himself 180 degrees in the opposite direction, of throwing the baby out with the bath water. Dale, you go right ahead and setup Utopia, Inc - and it should be fun to watch you puzzle out why Utopia Inc draws only dull, mediocre drones who bought a calendar at age 18 marking their path to retirement. For the few who aren't counting their days to retirement, you're likely to see true Hippies who love getting stoned in the bathroom and generally being fun and vivacious and fun to be around, and who's philosophy in life is to do as little real work as humanly possible while still drawing the pay check. As for me, I'll likely end up in one of those evil corporations who have the audacity to reward their workers equal to their contribution and abilities - and I'll be thinking of you as I deposit my bonus check in the bank. :-0

Posted by Mike Spille in General at 20040513 Comments[6]
20040504 Tuesday May 04, 2004

Home Improvement - Expanding the Cathouse As promised, here as some pics of the new and improved cathouse. As you can see I really have been very, very productive in my time off! Click on the pic to see a larger more detailed version.

  [image]

  [image]

The second picture perhaps deserves a bit of explanation. In the foreground on the right we see Cats Bane, a fearsome dragon Kristina and I obtained at great personal danger from the wilds of New Jersey. As you can see Bane is ready to swoop in on any unsuspecting cats who might dare lounge on the Cat House. A better view of Cats Bane is shown below:

  [image]

I'm sure you can see now why the cats fear him so and go to great lengths not to anger him. For one thing, they've seen what happened to the paratrooper who tried to take Bane out once.

And yes, I know I'm strange :-)
Posted by Mike Spille in General at 20040504 Comments[2]

20040501 Saturday May 01, 2004

Rediscovering Life So it's been one month and six days since I bid a not so found goodbye to SIAC (I was going to say this in French but was sure that I would botch it and Nathalie would punish me endlessly for my transgression; then again we haven't heard from Nathalie since the $10MM landed, so maybe I should've gone for it). Anyway, I gotta say that so far, it's been one helluva good ride.

Most surprising for me is that it's been one of the busiest times of my life. As I've commented privately to a few folks over the past month, I seem much more productive unemployed than I ever was working for someone. The difference seems to be personal interest and motivation - I'm doing things that interest me and I think that are worthwhile, as opposed to doing what someone else thinks is important. I realize now that a great deal of my corporate life was spent either doing things I didn't believe in, or trying to convince people that such activities weren't very worthwhile to begin with. Now, for the most part, the only arguments I have are with myself (and boy, can those be blistering!), and any mistakes made along the way point only back to me. It's a liberating experience.

In the past month, give or take, the following items have seemingly flown by, with lessons learned along the way....

13 Blog entries, three quarters of which were pretty meaty. The death of XA - well, not really, but at the very least the death of XA articles and code by me. I think I would be very happy not to hear those two letters in combination for a couple of years at least :-) On the above note, the last gasp of PyraLog. It barely made it to 0.1 Alpha 2, and there I suspect it shall linger 'till the end of time. An on-going relationship with the Groovy language (don't worry Kristina, it's not that kind of relationship, despite what people may tell you). I haven't written much about it, but I will soon once some of the decks are cleared again. The on-going growth of EmberIO. I am extremely satisified with the progress that I'm making there, and most gratified by the positive comments and encouragement I've received so far. A few more alpha releases are in order to flesh out some important details and missing features, but I think overall a solid beta should be done by a late June/early July timeframe. A big shout out and even bigger thank you to James Strachan for his great feedback on EmberIO while he integrates it with ActiveMQ. Learning open source environments and tools from the ground up. Sourceforge, CVS, common open source releasing practices - all new to me, believe it or not. Overall it's been fun to learn something truly new and different from the old stuffy corporate environments. If only Sourceforge weren't so flakey it would be even more enjoyable. The new kick-ass Sony VAIO laptop that Kristina gave me for my birthday. Two weeks into my voluntary leap into unemployment, my bride gifted me on my Birthday with a Sony PCG-K17 notebook with all the bells and whistles. It's become my primary development platform, with the big old PC reserved for graphic design work and server testing. This incredibly fantastic gift prompted me to upgrade the old home network along the way as well, so now I'm blazing along on a 16-port switch instead of a bunch of creaky old 10MBit hubs (of course, I mostly use the 'top in wireless mode, but still...). As a side benefit I also get to borrow one of Kristina's Mac's when she's off to work or school, so I'm learning OS X from the ground up as well. So far I can tell ya that OS X blows the doors off of any other OS environment I've used to date. Pounding the pavement and job hunting. Of course it can't all be fun and games - eventually I do have to nail down a permanent job. I've trundled through endless job sites, fielded a pretty decent batch of e-mail from potential employers, and gone out to contact others directly. Along the way, I've been on the most varied and interesting job hunt of my life. I've travelled to strange and distant cities far away, I've met several individuals in the local coffee house. I've penetrated some of the biggest companies you've heard of - and been kicked out by the same companies rubbing my ass in pain. I've interacted with jobs that are no more than nascent ideas and no money, small companies running lean and mean today but looking to grow, several solid medium side companies, and the aforementioned juggernauts. So far, no luck, at least on the perm side. But some of them do deserve at least honorable mention...

As I alluded to, I've been on a couple of interviews where I was judged lacking and metaphorically thrown out on the street by some corporate goons. My favorite therein has to be what I call the "Puzzle and Big Brain" interview. A group within a large corporation conducted a very Microsoftian interview where they didn't care what you did, or what you knew, or how well you develop software in practice. What they cared about was how well you solved Logic Puzzles, and how well you could handle arbitrary and complex design questions on your feet. In other words, they wanted one who was good with Puzzles and one who generally has a Big Brain.

I was asked "The ATM problem" - in a nutshell, how would you design an ATM? What the interviewer was getting at was pessimistic vs. optimistic locking, and the interviewer assumed that by saying "ATM" I would immediately understand what was being asked and zero in on pessimistic vs. optimistic. Well, as it turns out I didn't know that he was driving at that, and it took many tortuous minutes to get down to it. I mean hey, someone says "Design a bank ATM for me" and then just looks at you. What do you do, hotshot - What. Do. You. Do? Well, for me, mostly I mumble and say "Huh?" alot, apparently.

I was asked by the same person to draw out a design for a Java-like Garbage Collection system in as much detail as possible. It didn't matter whether I knew anything about garbage collection technology or state of the art practices - it was assumed that any intelligent human being could work through this problem and come up with clever answers on their feet, and thereby demonstrate their thought processes. Mostly I demonstrated that I can't design an efficient garbage collection system on my feet in 20 minutes time (Oh, forgot to mention - you're weren't allowed to design a bad GC that worked - it had to be a good GC that worked :-).

Another guy asked me to write a parser - in as complete code as possible - for a predefined arithmetic grammar. Now I know grammars, I know parsers and lexers, I've written alot of them. And I basically froze. I'd been in this little conference room for hours, wasn't really able to get around and stretch my legs or anything, and I froze. It didn't help that he insisted on code, and on code on a whiteboard no less. I don't know about you, but I can't write more than 10 lines of code on a whiteboard. So I froze. Oh, I could generally explain precedence and recursive descent vs. stack based, but I just couldn't code it on the spot. As a side prize, there were two problem he posited which I could describe most completely via recursive solutions - which he rejected. He doesn't like recursion, so I wasn't allowed to take that route.

He also hit me with a logical puzzle - in his words, well you're not doing so well with code, so let's try puzzles instead. So you've got a 300 page document, you drop it on the floor by mistake, and one page goes missing. The pages are numbered, 1-300, now what's the quickest way to determine the missing page? I gave him several obvious brute-force mechanisms - "No, those aren't efficient, keep trying". Out of frustration and perhaps a bit of claustrophibia being couped up in this little conference room, I mumbled "Well, you could probably use the page numbers somehow but I'll be damned if I have the stamina to figure it out". "Well, yes" he prompted me, and egged me on until out of fear he would keep me in that room all night I went up to the whiteboard scribbled numbers and sums everywhere and figured it out. But even though I kinda-figured it out, with a big hint, I wouldn't exactly call it a victory.

Lastly, an interviewer postulated a system with 4K of RAM, and then hit me with design questions and tradeoffs within that limited space. How would you model this kind of data structure, how much memory would it take? What if I changed it this way? What alternatives are there? What are the plusses and minuses of those approaches? In all the interviews spanned several very depressing hours.

I don't know about you, but I don't answer these sorts of answers well on the fly. I don't enjoy logic puzzles, and I think it's rather silly to ask someone to complete a full design for something as complex as GC in 20 minutes or less. The only questions I did well on were the one with the 4K of RAM - the interviewer was the youngest and most casual of the bunch, but he formed questions in a way that bounded them so they could be answered reasonably in the time allotted (and I'm uncommonly atuned to size vs. speed issues). The others just assumed that you could figure out puzzles with lightening speed under pressure, or casually designed things like GC systems in a few minutes in your free time. Worse of all, they would ask a question with an absolute minimum of words, and force you to drag more information out of them (and most often they would just say "not allowed" or "that doesn't matter"). It was the most antagonistic, useless, and personally humiliating interview I've ever been on. Needless to say, I was not asked back for a second round :-/
Locking down temporary sources of revenue. Despite my failure to secure permanent employment, I've had a bit more luck in the freelancing department. I'm working out a couple of part time freelancing gigs, mostly working from home, which supplement my savings sufficiently to take a big wad of time pressure off of me. Upgraded the Cat House. No, no, it's not what you think! This is a house I built for our three cats (a picture of the original cat house is here). Originally about 5 feet tall and 4 feet wide, it's now over 6 feet tall with additional levels. I'll post a new and improved picture soon for any cat freaks out there. The funny thing about the cat house is that it was built entirely from wood thrown out in my neighborhood - there's always NYU kids moving out around here and they thrown out things like almost-new wooden futons and similar furniture. So yeah, the cat house is financed almost entirely by spoiled NYU students. Being smart enough to quit with sufficient savings to not feel pressured in finding a new gig right now. Can't say much more than that. I went out and bought some ancient CDs that used to be my primary coding music - some old Prince, some old Def Leppard, some old Queensryche. And man, was it a fabulous idea. All I can say is that you cannot truly describe coding to Queensryche's "Operation: Mindcrime" - it is something that can only be directly experienced for yourself. Taking time to just be. Some days I take some time off from all of the above rigamarole and just rollerblade around NYC for a few hours. Bopping around the city on nothing more than 80mm wheels with an MP3 player for company can do wonders for centering one's self and taking in the world around them. When the weather isn't amenable, or I'm just not up to entering into death duels with taxis and NYC bus drivers, a simple walk for 5 or 6 miles can suffice just as well. Sometimes I'll watch a movie or two, occasionally I'll stay up 'till 4 in the morning, other days I'll veg out with a few video games for awhile. Some days I wake up at 7am for no reason and start coding, others I'll wake up at 11am and just lounge in bed for awhile. Sound self-indulgent? Not really - it's much more about bleeding off years of accumulated stress, and working with your internal rhythms instead of against them. Some days I can work 18 hours and feel exhilarated afterwards. Other days I'm sluggish and will veg on the couch. Sometimes I work in spurts, take a nap, work some more, break and review e-mail, break again and make a big dinner, work some more, etc. The rare times I need to get somewhere I'll take the subway, but otherwise there's no need for mass transit or rush hour crowds. If I need to be in mid-town at 2, I'll most likely walk the 50 blocks - it's fun, and good for me besides.

And here's the funny thing. Looking back it seems like I've wasted an awful lot of time - roller blading, sleeping, goof around, walking, playing games. But I've accomplished alot more with this lifestyle than I ever did at "work".
So the above has been my month of April, 2004, in a condensed format. Yeah, I'm still looking for the dream job, but there's no rush. I realize I can't do part time freelancing from home forever, at least not in the East Village of NYC, but for now it suffices. And sometimes, to me, that's what life is all about. It's not always about going for the gold, shooting for the stars, working in a frenzy towards that Liquidity Event or the next bonus cycle. Sometimes you Just Need to Be, occasionally it's better to coast then accelerate at full speed, and every once in awhile, just to suffice is nice.
Posted by Mike Spille in General at 20040501 Comments[16]
20040426 Monday April 26, 2004

EmberIO - Dispelling NIO Myths Apparently I've gotten XA stuff completely out of my system - and now I'm in a full-blown NIO obsession to compensate :-). I'm talking about EmberIO, of course, which as I briefly reported yesterday has been released on Sourceforge at version 0.3 Alpha. This blog entry will talk some more about EmberIO's strengths, more about the vision behind it, and try to show where EmberIO succeeds where other similar attempts have failed in the past.

To review, the main high-level features of EmberIO include:

Support for many different threading models, and for blocking and non-blocking I/O. Automated NIO buffer management, including automated support for partial reads and writes in a non-blocking environment Management of one or more NIO Selectors for performing multiplexed I/O. This management is invisible to the application code. This is new to EmberIO as of 0.3 Alpha. Ultimately, to provide support for both NIO-based I/O using NIO channels, and non-NIO I/O resources such as multicast sockets. Differences between NIO and non-NIO resources will be completely hidden from application code by default. Note the use of the future tense here ;-). EmberIO doesn't do this yet, but the stage is now set for me to provide such support easily. Different I/O models give different performance characteristics, with each model representing a tradeoff between resource utilization, latency, throughput, and the physical size of data being read or written. Working extra hard to be both high-performance and correct - so you don't have to :-)
The biggest bang for the buck you'll get out of EmberIO is in the buffer management, built-in support for a variety of threading models, and from generally getting rid of the pain usually associated with NIO. Also keep in mind that while one of the initial motivations behind EmberIO is to make using NIO easier, it is being expanded to handle non-NIO I/O resources as well. EmberIO is moving towards being a general low-level I/O sub-system that puts a single consistent interface on top of disperate I/O resources and threading models.

EmberIO Threading Models

One of the most interesting EmberIO features is the threading model support - it offers a boatload of them, each of which is targetted towards different I/O scenarios. Almost any scenario may be configured to be used in either a blocking or non-blocking manner (with a few exceptions where the model constrains you to one or the other). On top of this, EmberIO supports both traditional blocking Java I/O semantics (which I'll call BIO here - thanks to James Strachan for keying me on this term!) and newer NIO asynchrous/multiplexed/non-blocking semantics (which I'll just call NIO).

A non-exhaustive list of threading models include:
BLOCKING_ACCEPTOR
Creates a dedicated thread for handling accept() calls for a socket server in a blocking manner.

NONBLOCKING_ACCEPTOR
Handles socket server accept() calls in the traditional NIO manner e.g. by using the OP_ACCEPT interest bit in a Selector. For most people this is more of a pain in the ass than anything else, and BLOCKING_ACCEPTOR would be more appropriate.

USE_RWP_POOLS
Uses seperate thread pools for handling READ, WRITE, and PROCESSING events. Each of these will be handled indepedently (at least from a threading perspective). Most useful if you anticipate doing large READs or WRITEs or both. Any of the *_POOL variants aggressively use a PROCESS and WRITE FIFO queues to decouple read/write events from the events themselves and actually performing the physical I/O (e.g. READs and WRITEs happen completely asynchronously).

USE_ONE_POOL
Uses a unified thread pool for all events. Often useful if you have small READs and WRITEs and want to reduce latency significantly.

DEDICATED_READER
Sets up EmberIO threading and I/O to be performed in like old-style BIO. Individual sockets (or what have you) are setup with dedicated blocking reader threads, and writes are performed in a blocking manner directly to the I/O resource. The DEDICATED_READER model is interesting because it opens the door for supporting non-NIO resources. With DEDICATED_READER in place and bypassing the NIO Selector, EmberIO is in a position to deal with any old BIO-style socket-like component - and to make such a resource look the same as NIO resources from an EmberIO user's perspective.

SELECTOR_READER
Performs READ operations within the context of a Selector's main event loop thread. This forces all reads to be done in a non-blocking manner (after all, we don't want to block our selector loop!) and can only be used for NIO-enabled I/O resources (e.g. right now, only Sockets).
The above models are configurable per connection type. So, for example, you could configure one type of connection using the DEDICATED_READER strategy for the lowest possible latency, and another type of connection could be configured for USE_RWP_POOLS where many connections are expected with variable I/O sizes, some of them potentially large, and meanwhile setup all your server sockets using the BLOCKING_ACCEPTOR strategy.

The models tie into how the default I/O events that EmberIO supports: READ, WRITE, PROCESS, and ACCEPT. Leaving ACCEPT out of the picture for the moment, EmberIO is setup so that data flows in the following manner:
READ events trigger physical reads into ByteBuffers (which are under control of the app). When a READ gets a full object, it stuffs on the object on a the PROCESS FIFO queue. These READ events are generally triggered by an NIO Selector. PROCESS events get triggered when something gets put in the PROCESS queue. This is independent of the Selector. WRITE events get triggered when a user calls write() on a ReadWriteEndpoint. What happens here is that we stick the object to write on a WRITE FIFO queue, and then fire the WRITE event. In practice, if we're in non-blocking WRITE mode we try the write first, and if that doesn't work (or is incomplete) we then add OP_WRITE interest to the endpoint and wake up the Selector to deal with it.
EmberIO takes advantage of the fact that I/O and PROCESS often flow in this sort of manner:
Server reads a request Server processes request Server writes a response to the request
Since this is so common, EmberIO is setup to try to accomodate this directly. Handlers for various events are ordered in READ-PROCESS-WRITE order (for use when a single common thread pool is used) and EmberIO tries to execute events directly before getting the Selector involved. So it's common for EmberIO to get triggered for a READ, and then perform the READ, the PROCESS, and the WRITE in one chunk - with the Selector involved only in the initial READ triggering. Adding in greedy operations (described below) means we can burstily do "N" such operations without involving the Selector.

Of course, the above does not always hold true - some of the BIO-like models circumvent things like the WRITE FIFO queues entirely. Different needs, different strategies, different trade-offs.

Related Features

In addition to the various threading models, EmberIO includes a few other features to help the developer tune the model towards the best balance of resource usage/throughput/latency that matches their needs. Most of these are configurable, but a few are hard-coded into the library to work around known NIO (or plain BIO issues) or to boost performance.

One thing to keep in mind about EmberIO is that all features are to some extent intertwined, and some features negate the use of others. As one example, use of DEDICATED_READER effectively negates the greedy operations feature, since DEDICATED_READER channels bypass the Selector entirely. Keep this in mind when you're trying out the various EmberIO configruations, and above all use your head!
Greedy operations.
EmberIO inherently is pretty aggressive in its use of threads, but at the same time naive approaches to NIO tend to lead to excessive thread switching. To make matters worse, Selector performance isn't always what it could be. Because of this, greedy operations were born. Greedy operations basically enable a thread to do "N" units of work once it has control. For example, a READER could be configured with greedy ops set to "20", which would mean that the READER would try to read up to 20 objects before relinquishing control. Judicious use of greedy ops can significantly boost your server's throughput by doing work more outside of the Selector, and minimizing use and interruption of the Selector.


Throttles.
It's fairly common for I/O to run away out of control in Java applications. This is true in particular when you're using asynchronous READ and WRITE operations. You can end up in a situation where either your I/O threads are eating all of your CPU time, or just as bad, your PROCESS or WRITE queues start filling up monstrously and you start running out of memory. The initial implementations of EmberIO didn't address this problem at all, and as result it was common under my stress tests to end up with a WRITE queue containing several hundred thousand objects waiting to be written, for the network to be saturated with a storm of packets, and for PROCESS workers to be starved (or at least seriously undernourished). The end result was a serious unbalanced utilization of available resources, out of memory errors under long lasting heavy loads, and latencies in the tens of seconds. Yuck.

As of 0.3 Alpha, EmberIO supports throttles to address this problem. You can configure EmberIO to throttle READs or WRITEs so that they shut off if either the PROCESS or WRITE queues fill up past a certain threshold.

On the READ side, for most models we turn off READ interest if the PROCESS queue passes this throttle threshold, which effectively means we stop reading. This state persists until a restart threshold is reached - basically, the PROCESS queue is drained sufficiently, at which point we re-inject READ interest. If the DEDICATED_READ model is used, the job is much simpler - we just block the READ thread until the restart threshold is reached.

On the WRITE side, we physically block a thread trying to WRITE in most models if the WRITE queue threshold is exceeded. One gotcha here is that we can't do this if the writer is an EIO thread, since we'd be potentially deadlocking ourselves. To get around this, right now EIO will only block on non-EIO threads (this is done by checking the ThreadGroup of the writing thread). A more sophisticated model may be implemented in the future, since this current one is a bit white-trash and naive. For the DEDICATED_WRITE scenario, we don't do anything special right now - we assume the I/O resource itself will throttle us, and in case no WRITE queue is used, so there's nothing to fill up. This should probably be changed in the future, to avoid flooding the network.

Thread Priority
EmberIO lets you set worker threads' thread priorities along with the pooling strategy you're using. Changing such priorities can have very dramtic results for certain situations - and can also lead to massive thread malnutrition if you're not careful! First-class support for thread priorities are provided in recognition that sometimes not all threads are created equal.

Read and Write Optimizations
EmberIO has a bunch of optimizations to try to efficiently get data in from your I/O resources and back out again. In particular, we try reads and writes a few times if they are incomplete, even in non-blocking mode. This works extraordinarily well because most JDKs and TCP/IP stacks will often return/write just one byte, even if all the data (or buffer space) you need is there. Quite often a subsequent read/write immediately after will snarf/write the rest of the data.

Auto Management of Blocking Semantics
EmberIO auto-manages the blocking configuration of your channels so you don't have to. More importantly, it does this in a safe manner that avoids the dreaded IllegalBlockingModeException. While you should generally try to use non-blocking I/O for everything, this isn't always a feasible option, and EmberIO makes sure you can write blocking code within it just as easily as you could in the old BIO model.
The above features, in combination with the various threading models, are designed to get the most out of the Java I/O system, and to allow users to easily change the runtime model to match their needs. And while doing so, dispelling some myths about NIO along the way.

Using NIO Intelligently

NIO has gotten a pretty bad rap in a number of circles. A number of people have gone out, written some NIO code, and concluded: "NIO sucks donkey wood, man!". Invariably the critiques have boiled down to a few observations:
Latency is too high in NIO - the cost of the Selector and other threads are so much higher than with BIO that it's not worth it. Threads are free! Who cares if BIO requires a thread per connection? I tried to use NIO and started getting all sorts of exceptions - IllegalBlockingModeException, CancelledKeyException, EOFException, YoMamaWearsArmyBootsException. NIO sucks! I switched to NIO and my server throughput plummed by X%
To some extent these critiques are true - programming NIO directly is more difficult than BIO, and the "obvious" way to code NIO is highly inefficient. But the real problem is that most people who have tried NIO have only tried it the "obvious" way, and never delved very deeply into tuning their code to how NIO really works under the covers.

I've looked at several people's NIO solutions, and almost universally they make the following mistakes:
The OP_ACCEPT interest op is there, so they assume they must use it. So each new connection requires you to pop out of the Selector, find the right thread for your ServerSocket, and then call accept() on it. And yeah, this is slow. The solution to this slowness is simple: don't use OP_ACCEPT, just stick your ServerSocket accept() in a dedicated thread, and watch accept latency disappear. EmberIO supports this directly and transparently with the BLOCKING_ACCEPTOR strategy. They lock themselves into one threading model. They either always do non-blocking reads in the Selector thread, or they always do reads from a seperate thread gotten from a thread pool. They tend to always do writes directly to the channel. Then they step back and look at the results - and they see that sometimes their hard-coded model model works well, and in other cases it's not so good. They dither back and forth for awhile and conclude that it's not worth it. EmberIO doesn't lock you into any specific thread model - you can go thread-crazy with seperate thread pools for everything, or you can do non-blocking READ from the Selector, or you can piggyback items onto a single thread pool, or you can go BIO and have a dedicated thread per connection. And you can configure each type of connection you're using differently, so you can have a mix at runtime. They use only one Selector. Why not make your code configurable and use "N" Selectors? Why assume that every Selector implementation will scale from 1 connection up to thousands? EmberIO automanages Selectors for you so that you never even have to see them, and can use just 1 or as many as you think is appropriate. They get whacked by Socket read and write realities. Most people don't seem to realize that Java sockets really love to deal with just one byte at first, and then open the flood gates immediately afterwards. For example, if you do a read quite often your socket will give you just one byte, or just a few - but a read immediately afterwards will give you a buttload of data. Likewise, writes often will write only a byte or two - and another write right afterwards will blam out a couple of K. The problem that many people run into is that they code their non-blocking I/O code rather literally. They try to the read/write, and if it didn't complete they chuck the socket back onto the Selector with the appropriate interest ops set. The end result looks like this for reads in many implementations:
OK, you're sitting in select(). You pop out of Select, with OP_READ set to ready. You delegate to some thread. The reader thread sets up their ByteBuffer with N bytes, does channel.read(). Gets 1 byte - nuts, that's not all my data! Throw the thing back on the Selector since we didn't get all the data. This requires you to wakeup the Selector with wakeup(). Selector goes back into select(), immediately pops out with OP_READ ready again! You delegate to some thread - again. You do channel.read() again on the pre-setup ByteBuffer. _Now_ you get all the rest of your bytes!
So each READ, even for small amounts of data, ends up requiring two selector wakeups, two thread delegations, and two actual read operations. Yuck!

Writes follow a similar pattern. Except that they first add WRITE-interest, then wakeup the selector to get that to "take", then end up going through the cycle twice just like for reads. EmberIO was coded with full awareness of this odd quirk of sockets, and if a non-blocking read or write does not complete on the first try it tries it again immediately. Knowing what you know now, you won't be surprised to hear that this tiny optimization boosts throughput by more than 50%. At best, they do one operation at a time. Once a read or write is successful - they bolt right back into the Selector. Argh! Why do that? EmberIO supports "greedy operations", so it'll try "N" operations in a thread before popping you back into the Selector (with "N" configurable, and of course it gives up if it just can't read/write/process/whatever). This little optimization again boosts throughput by about 25% They fight against NIO instead of bowing to reality. The threading rules for Selectors and blocking modes and the like drive them crazy, and they howl at the moon in frustration as they keep getting various Exceptions. I'll be the first to admit that threading rules for NIO are a pain in the ass, but I'm not going to pull out my hair and rage around the room because of it. NIO is what it is, and EmberIO is coded with NIO realities in mind. It deals with all the rules so you don't have to.
As I said, lots of people have tried out NIO and gave it a big stinky thumbs down - but that thumbs down was typically based on writing a couple of hundred lines of code (or even tens of lines!) and giving up early when the results weren't encouraging. EmberIO approaches all of this from a pragmatic perspective - I want to do some I/O. I want it fast, I want it correct, and I want it configurable to varying I/O needs. Increasingly, EmberIO is actually become NIO-agnostic - if you want Selectors and lots of threads, it'll give you that. But if you want something closer to the old BIO model, it will give you that, too. And most interestingly (to me at least), it can also give you compromises somewhere in the middle between BIO and super-aggresive NIO. I'm sure a group of people would say that EmberIO "cheats" - it does little more than encode a number of tricks to work around NIO limitations. To which I'd say "Yeah - and so what?!". The important point is that it gets the job done. And in many scenarios EmberIO can give you the benefits of NIO - direct buffers, easy non-blocking I/O, configurable thread pools - with almost no sacrifices in throughput and latency. And it scales - you can do servers on little machines with limited resources using a pooled non-blocking strategy, and scale it up to bigger loads and hardware by configuring more Selectors, changing the I/O strategy, changing the threading model, tweaking parameters like the throttles and greedy operations. And if the situation calls for it, you can even set it up to do plain old BIO. In the end, with EmberIO the point isn't a religious argument of NIO vs. BIO, or blocking vs. non-blocking, or wars over different threading models - or even about bitching at Sun. The point is to give you options and let the user decide what works best for their situation.
Posted by Mike Spille in NIO at 20040426 Comments[11]
20040425 Sunday April 25, 2004

EmberIO 0.3 Alpha Released The EmberIO library, version 0.3 Alpha, has been released. Unlike previous versions this is a fairly real release - in the sense that it's now hosted on sourceforge.net, and I actually followed a semi-automated release process :-)

Go to the project page on sourceforge for details. Downloads are available in both .tar.gz and .zip format. Under "Latest File Releases", click on "EmberIO_0.3_Alpha" to see the various file options.

I'll have a blog a bit later on details of the release, and more general ramblings on NIO. Right now I'm just too tired....
You can see the README and the changelog for this release here.

P.S. OH, I almost forgot to mention that this version of EmberIO is really, really super flexible :-)
Posted by Mike Spille in NIO at 20040425 Comments[0]


You are viewing a mobilized version of this site...
View original page here

Mobilized by Mowser Mowser