My Background

I have a background as a Java/J2EE programmer and application server administrator. Since you’re reading this post, you are probably aware of the issues the Java platform has faced during the slow demise of Sun Microsystems. There is a sense that Oracle is taking the reigns to complete a long overdue release of Java7, but no concrete terms have been established. After such a debacle, it’s hard to imagine Java being a first choice for any developer with a perspective on the field.

With that uncertainty in mind, I’ve explored a variety of other languages over the years, including Groovy, Python, C#, and Erlang, completing projects in several. I assume most Java folks have taken this route to remain competitive in the market.

I founded the consulting business hoping to market those broad skills as a general troubleshooter, architect, and developer. Someone who could arrive at an organization and suggest and implement some best practices. My focus was efficiency: cutting development costs by utilizing commoditized resources and streamlining inefficient processes.

However, I didn’t have much experience in the business and marketing world. Developing people skills can be exceptionally difficult for someone who has spent most of their professional years behind a computer screen in a cubicle. After several months of building a business, I truly believe the sales process is phenomenally more difficult than completing the actual work.

The rest of this article details a few of the key issues I’ve encountered while learning (or at least attempting to learn) the art of business building.

Start By Leveraging Your Existing Network

A family friend once told me “it’s not what you know, it’s who you know”. A different friend mocked this suggestion, believing the commentor was lazy and only capable of succeeding because of his ability to exploit relationships. Looking back, the friend who mocked has excelled in the construction business by leveraging his experiences in prior jobs.

As Americans, we hold dearly the concept of self-determination. Hard work and a go-getter attitude will open every door. However, it is impossible to discount the impact of mutual effort in any significant undertaking. Human beings are social creatures who build networks to better confront the challenges of nature.

I decided to start the business after moving across the country, back to the medium sized city where I had lived and worked for two years before moving to California. I assumed that my old colleagues from so many years before would be a great source for new opportunities. Unfortunately, this was an extremely egotistical notion, believing that people I had known as acquaintances from so many years before would jump at the opportunity to help me succeed. As if they didn’t have lives, careers, and families of their own. To this day, I’ve only managed to have lunch with one of those old co-workers. This person was also a business developer and probably more obligated to relationship building; otherwise, I honestly don’t think he would have taken the time.

My first stable client was my former employer from California. Even with a country separating us, the level of trust was much more established between people with solid, recent memories and associations. A former college roommate gave me an opportunity after reminiscing about college experiences at a wedding.

If you plan to freelance, you will find much more success engaging your current contacts, friends, and family.

Extending Your Network

If you find yourself without a strong network to draw from, you’ll need to get out and start building one. I started by attending “networking meetings”, found on sites like Meetup.com. Some are more informal than others. The most famous (and probably infamous) of these is BNI, a group that charges an annual fee and has a strict schedule. Their central premise is that any new business opportunities should be channeled through group members. The members are paying, both in time and money, for the network. I attended a few of these meetings, both with BNI and PRO (a group local to Richmond).

Unfortunately, I found that most of the services offered by the members were business to consumer oriented. For example, there were a lot of insurance sales people and realtors. These groups make sense for people in those industries, but my technology services were geared more towards enterprise. It might have been possible to sell some small to medium size businesses, but I would strongly recommend anyone following this approach to look for more business-to-business oriented groups.

In technology, users groups tend to be better sources of contacts. For instance, I’ve met several folks at Linux User Groups who not only share interests, but can provide connections into the companies where they work.

Prioritizing

Another big issue is what I term “thrashing”. In technology terms, this indicates a machine that is spending more time managing resources than accomplishing actual work. In the consulting world, I define “thrashing” as continuously developing efforts that never turn into billable hours. Many tasks could be included in this category.

Unless you are a full time business developer, attending networking events that are inappropriate for your offered services is huge time drain. If you decide to approach these groups, try to talk to some members before spending the full hour to two hours, travel included, required to participate.

Be very wary when approaching new clients. In today’s economy, people are looking to squeeze dollars. I’ve found that a strict limit on the amount of non-billable development time is absolutely necessary in a new engagement. Presenting a good strategy and starting point quickly in the process, along with clear expectations, will save a lot of potentially wasted time.

Don’t get stuck with a large amount of small projects. I think two to three clients, certainly as a sole proprietor, should be a limit for any responsible consultant. If you are finding you wind up and down a large amount of business within two or three weeks and you spend more time developing relationships than billing, then you should reconsider your strategy. Of course if you have a process or product geared towards that strategy, you might find more success. However, trying to take on too much is a disservice to both you and your customers.

Find a Niche

The roles of contract programmer and systems administrator are largely commoditized. International freelance websites and large full service IT shops have largely cornered these markets. As a consultant, you need to find a specialty that is not widely available and will greatly impact the bottom line. Examples might be reducing redundant processes or support costs.

If you are having problems defining that niche, find ways to communicate with business leaders in your field of expertise. I spent several years working in healthcare, mostly around HIV and the associated research. A SCORE counselor suggested I contact local hospitals to determine where there are lacking in service providers. Of course, you will need to define what you plan to offer, but after a few engagements you might determine a developable option.

Once you’ve determined that niche, target your marketing and development efforts. My business offering is very general, not targeted to efficiency or cloud processes, and dilutes my pitch behind unrelated services. To date, I haven’t turned down any clients, even when the work doesn’t relate to my business. Starting to do this has been difficult to consider, certainly with rent bills due, but should strengthen my brand and expertise over the long haul.

Accept Defeat

Stubborn people have a hard time accepting failure and learning from their mistakes. I certainly fall into this category. However, I am now searching for contract work through recruiters so I can begin to retire clients that don’t fit my business model. The long term goal is to reduce my client load to one while I redesign my web site, business cards, and sales strategy.

I hope some of these ideas will help focus your business initiatives, feel free to leave a comment if you have any additional ideas.

Here at Open Initiative Consulting, we recognize the benefits of these new technologies, but have also encountered a few problems. One problem is determining when a unit of work is complete. A work unit represents the combined total of segmented independent processing tasks, that run on separate machines. These processing tasks have no direct communication with a “parent” once they have been allocated, but can record internal state on their progress. After all of these processing efforts are complete, the results are aggregated at a single “reducer”. Normally, we could record the number of children nodes when they are spawned and consider the unit complete once all children had reported. Unfortunately, our requirements were not so simple; children have the ability to generate more than one additional child. Here is an example (notice the third child on the right):

At first glance, we thought about complex algorithms to trace chains back to the root to determine how many possible children were still processing. However, by tracking the number of children each child spawns, we can use simple math to calculate the “weight” of each independent child against the composite work unit. From our previous example:

Now we can do some simple math to determine the percentage of completed work as each child completes:

This calculation does carry some caveats. There is a certain margin of error while performing floating point division in most programming languages. In java for example:

1
2
3
4
5
6
7

public class Test {
    public static void main(String[] args) {
        double calc = (1.0 / 3.0 / 2.0) * 100 + (1.0 / 3.0) * 100 + (1.0 / 3.0) * 100 +
            (1.0 / 3.0 / 2.0) * 100;
        System.out.println(calc);
    }
}

produces “99.99999999999997″. To avoid miscalculation, we multiply by 1000, round the result, and divide by 1000:

1
2
3
4
5
6
7
8
9

import java.lang.Math;
 
public class Test {
    public static void main(String[] args) {
        double calc = (1.0 / 3.0 / 2.0) * 100 + (1.0 / 3.0) * 100 + (1.0 / 3.0) * 100 + 
            (1.0 / 3.0 / 2.0) * 100;
        System.out.println(Math.round(calc * 1000) / 1000);
    }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

def jsonarray(array):
    '''create and encode a json array'''
    val = (jsonvalue(i) for i in array)
    return "[" + ",".join(val) + "]"
 
def jsondate(date):
    '''string format date'''
    return jsonencode(date.isoformat())
 
def jsondict(d):
    '''create and encode a json array'''
    res = lambda k,v : "%s : %s" % \
        (jsonencode(k), jsonvalue(v))
    res = ",".join((res(key, value) \
        for key, value in d.iteritems()))
    return "{%s}" % res
 
def jsonint(i): return str(i)
 
def jsonsimple(i): return jsonencode(str(i))
 
_jsonhandlers = {
    datetime : jsondate,
    dict : jsondict,
    int : jsonint,
    list : jsonarray,
    tuple : jsonarray,
}
 
def jsonvalue(value):
    '''handle json value conversion'''
    return _jsonhandlers.get(type(value), jsonsimple)(value)

and the XML:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

def xmlarray(array, key="result"):
    '''create and encode an xml array'''
    vals = "</%s><%s>" % (key, key)
    vals = vals.join((xmlvalue(i) for i in array))
    return "<%s>%s</%s>" % (key, vals, key)
 
def xmldate(date):
    '''string format date'''
    return date.isoformat()
 
def xmldict(d):
    '''create and encode an xml array'''
    res = lambda k,v : "<%s>%s</%s>" % (k, xmlvalue(v), k)
    res = "".join((res(key, value) for key, value in d.iteritems()))
    return res
 
def xmlsimple(i): return i
 
_xmlhandlers = {
    datetime : xmldate,
    dict : xmldict,
    list : xmlarray,
    tuple : xmlarray,
}
 
def xmlvalue(value):
    '''handle xml value conversion'''
    return _xmlhandlers.get(type(value), xmlsimple)(value)

Calling these methods is fairly straight forward in the templates. After determining the output type (I’ve been using the “Accept” header), here is a JSON example:

1

${h.jsondict(c.result) | n}

and XML:

1

<result>${h.xmldict(c.result) | n}</result>

As an avid unix user, the cron daemon has been an irreplaceable tool. The parameters are intuitive, with options to set the minute, hour, day of month, month, and day of week, that allow any pattern of scheduling. It is possible to build a similar simple scheduling system for an application; two methods add a schedule and retrieve current schedules. These examples rely on python and mongodb, but can be easily adjusted for a different language and a relational database.

Here is the scheduler with constant definitions for the field keys. I use the required “params” key as a dictionary holding the task configuration. This code of course doesn’t handle all cron options, such as ranges and step values, but should suit many tasks. If a value isn’t specified by a named parameter, it becomes a wildcard match, similar to cron:

1
2
3
4
5
6
7

_datefields = ["minute", "hour", "day", "month", "dayofweek"]
def schedule(user, params, minute="*", hour="*", day="*", month="*", dow="*"):
    '''add a schedule item to the database'''
    schedule = {Extract.USER : user, S.PARAMS : params}
    vals = [minute, hour, day, month, dow]
    for cnt, key in enumerate(_datefields): schedule[key] = vals[cnt]
    mongodb.schedules.save(schedule)

And the schedule retrieval code, which returns current entries. This code basically looks for a match on either the ‘*’ pattern or the current timestamped value:

1
2
3
4
5
6
7

def current_schedules():
    '''find schedule items that need to be run'''
    dt = datetime.datetime.today()
    look = {}
    vals = [dt.minute, dt.hour, dt.day, dt.month, dt.weekday()]
    for c, k in enumerate(_datefields): look[k] = { "$in" : ["*", vals[c]]}
    return mongodb.schedules.find(look)

Here is some simple client code:

1
2
3
4
5
6
7
8
9
10
11

def run():
    try:
        while 1:
            for schedule in current_schedules():
                _log.info("batch item at [%s]" % datetime.today())
                some_execution_method(schedule["params"])
            # try to wake up at one second after hour
            time.sleep(60 - datetime.datetime.today().second + 1)
    except KeyboardInterrupt: print("killed...")
 
if __name__ == "__main__": run()

Of course, this code isn’t thread-safe or “multiple-client-safe”, so running multiple clients is probably not a good idea.

There are several implementations of this strategy; of which, Hadoop seems to be the most respected in the open source world. Yahoo has fully embraced the project.

However, Hadoop does have some caveats. Configuring a cluster requires editing a multitude of XML files. Hadoop filesystem has to be installed. The master node is not fault tolerant. Typically the “slave” node locations are stored in file, though new ones can be added dynamically. Hadoop also requires jobs to be written in languages that will run on the JVM.

When evaluating Hadoop, we decided it was a little too complex for our first proof-of-concept for a client. Plus we want to use tools like pypy with our Python application to boost performance. But what was an alternative? Disco? Seemed reasonable. Unfortunately, messages had to be pushed via SSH to “workers”, making them difficult to organize within firewalls. And additional workers have to be added manually through the web interface.

Sometimes square pegs just won’t fit into round holes. So, what technology was round enough for our needs? We knew distributing tasks was naturally solved by queues; one went in, another went out, until the queue was empty. In the networked world, message queues can spread job information across machines and geography. AMQP allows advanced routing of these messages through exchanges and is widely supported by a multitude of languages. Message queuing systems are also designed to be highly fault tolerant and distributed.

And it worked pretty well. We used py-amqplib as a client attached to rabbitmq with some fairly basic code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96

from amqplib import client_0_8 as amqp
from dobie.extract import configval
import logging
import uuid
 
log = logging.getLogger(__name__)
 
class Connection(object):
    '''basic connection to mq server'''
 
    def __init__(self):
        self.host = configval("host")
        self.user_id = configval("user_id")
        self.password = configval("password")
        self.port = configval("port")
        self.connect()
 
    def connect(self):
        self.connection = amqp.Connection(
            host='%s:%s' % (self.host, self.port),
            userid=self.user_id, password=self.password
        )
 
class Base(object):
    '''shared code for publisher and consumer'''
 
    def __init__(self, queue, connection=None):
        self.connection = connection or Connection()
        self.channel = self.connection.connection.channel()
        self.ex = configval("exchange")
        self.queue = queue
        params = (self.queue, self.ex, self.queue)
        log.debug("queue [%s], exchange [%s], routing [%s]" % params)
 
    def setup(self):
        '''configure queue and exchange'''
        self.channel.queue_declare(
            queue=self.queue, durable=True, exclusive=False, auto_delete=False
        )
        self.channel.exchange_declare(
            exchange=self.ex, type='direct', durable=True, auto_delete=False
        )
        self.channel.queue_bind(
            queue=self.queue, exchange=self.ex, routing_key=self.queue
        )
        return self
 
    def close(self):
        '''close connections'''
        # TODO consider pooling
        self.channel.close()
        self.connection.connection.close()
 
    def __enter__(self):
        return self.setup()
 
    def __exit__(self, _type, value, traceback):
        self.close()
 
class Consumer(Base):
    '''consumer of messages'''
 
    def __init__(self, queue, callback, connection=None):
        Base.__init__(self, queue, connection)
        self.callback = callback
        self.cid = str(uuid.uuid4())
 
    def __enter__(self):
        def _callback(message):
            self.callback(self, message)
        Base.__enter__(self)
        self.channel.basic_consume(
            queue=self.queue, callback=_callback, consumer_tag=self.cid
        )
        return self
 
    def wait(self, count=-1):
        '''wait for messages. if count is set, only consume a limited number
           before exiting'''
        while count != 0:
            self.channel.wait()
            if count > 0: count = count - 1
 
class Publisher(Base):
    '''publisher of messages'''
 
    def __init__(self, queue, connection=None):
        Base.__init__(self, queue, connection)
        self.delivery_mode = int(configval("delivery_mode"))
 
    def publish(self, message):
        message = amqp.Message(message)
        message.properties['delivery_mode'] = self.delivery_mode
        self.channel.basic_publish(
            message, exchange=self.ex, routing_key=self.queue
        )

There are some potential problems with this code. Connections aren’t pooled and it only uses a “direct” exchange. However, it suited our needs.

We relied on dictionaries to transfer data. Since we planned to stick with python, we pickled them before sending them over the line, which rabbitmq was happy to accept.

There is one problem which we have yet to resolve: if a client is connected and processing messages, newly connected clients will only receive messages that are added later. Will post a fix if we figure it out.

Any web application that approaches moderate complexity almost always stumbles across a data transmission problem. Modeling relationships that need to be stored back at the server, such as parent-child or groups, with key-value pairs is not a natural fit. Anyone who worked with struts in the early part of the decade is very aware of this problem.

On the other hand, the DOM and more generally XML feature a natural method of designating these associations by embedding tags. For instance, in HTML one could code:

1
2
3
4
5
6

<div id="parent">
  <span class="name">jerry</span>
  <div class="child">
    <input class="name" value="joey"></input>
  </div>
</div>

which indicates “joey” is the child of “jerry”. Now imagine trying to transmit this relationship with HTTP parameters. This might work:

parent=jerry&jerrychild=joey

but deciphering such a statement on the receiving end would be complex and fragile. A document database like CouchDB might fit the task, but then you would have to create javascript code to generate JSON from the child elements. In other words, double code both the page generation and the javascript to “serialize” objects.

real world experience

At a recent client, we faced this problem in a dramatic fashion. The group was comprised mostly of front-end experts, who took XML data from a custom CMS and transformed it with XSL to produce their final content. They couldn’t write the server code to store the submitted data back into the database. And they wanted to create very complex relationships in their data models.

modeling data in the DOM

To mitigate this skill-set disconnect, we created a javascript library in jQuery to generate an XML document from embedded tags in the DOM. This library iteratively loops through a specified element and it’s children, searching for specific tags. If an element has the class “complex”, a tag is generated with the value from the “id” or “name” attribute before it’s children are processed. If an element has a “primitive” class, a simple element is generated from it’s “id” or “name” attribute and value. An attribute is created by specifying an “id” or “name” attribute with an “att” tag.

Following our earlier example:

1
2
3
4
5
6
7

<div id="parent" class="complex">
  <span name="name" class="primitive">jerry</span>
  <div name="child" class="complex">
      <span name="birth-order" class="att">1</span>
      <input name="name" class="primitive" value="joey"/>
  </div>
</div>

would produce:

1
2
3
4
5
6

<parent>
  <name>jerry</name>
  <child birth-order="1">
    <name>joey</name>
  </child>
</parent>

This data can be submitted (via ajax) to the backend for processing, updating, validation, whatever. You can download library here:

http://github.com/greyrl/relationalxml/blob/master/js/jquery.dom.js

You will also need the custom jQuery “class” and “util” packages. After importing the libraries, you can generate XML Documents or strings:

1
2
3
4

// documents
var docs = $("#parent").xmlGen();
// strings
var docStrings = $("#parent").xmlGen().innerXML();

If you want to send that data back to the server, you should also consider Base64 encoding.

caveats

If the id attribute for a group cannot be unique, this library does require a user to bastardize the DOM by adding the unsupported (at least on most elements) “name” attribute. However, we feel it is a small price to pay for flexibility.

The “development” portion of Selenium is a Firefox plugin that records user clicks, paths, and form entries as the user interacts with the browser. Normally, this “recording” is converted to a programming language (by Selenium) and tweaked for automated, continuous testing during web development. However, generating code from a selenium script opens up the world of flexibility provided by the underlying programming language. For example, we could use loop constructs to continuously click on a “next” button while searching for a specific piece of data. This data could then be stored in a database or file, even sent to a web service.

Here is a very basic example of such functionality:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

from selenium import selenium
import unittest, time, re
 
class sample(unittest.TestCase):
 
    def setUp(self):
        self.verificationErrors = []
        self.selenium = selenium("localhost", 4444, "*chrome", "http://your.site.here")
        self.selenium.start()
        with open("parts.txt") as f: self.parts = f.readlines()
 
    def testSample(self):
        sel = self.selenium
        for part in self.parts:
            sel.open("/database.html")
            sel.type("partnumber", part.strip())
            sel.click("go")
            sel.wait_for_page_to_load("30000")
            # look for a numeric pattern
            sel.click("link=regexp:[0-9]{7}")
            while 1:
                try: 
                    sel.wait_for_page_to_load("30000")
                    for i in range(1,8):
                        id = sel.get_table("//table[3].%s.1" % i)
                        status = sel.get_table("//table[3].%s.5" % i)
                        print "id %s, status %s " % (id, status)
                        if status == "READY": self._processHit(id)
                    sel.click("next")
                except Exception as e:
                    print "error or end of processing encountered"
                    #print e
                    break
 
    def _processHit(self, id):
        print "process %s" % id
        sel = self.selenium
        sel.click("link=%s" % id)
        sel.wait_for_page_to_load("30000")
        with open("result.txt", "a") as f:
            f.write("id %s, name [%s]\n" % (id, sel.get_table("//table[3].6.2")))
 
    def tearDown(self):
        self.selenium.stop()
        self.assertEqual([], self.verificationErrors)
 
if __name__ == "__main__":
    unittest.main()

Basically the order of operations is:

It probably isn’t a great idea to open and close the “result.txt” file for each result, but this is a pretty basic example.

Also, the translation from row to object is cumbersome and just doesn’t seem to completely fit. Object-relational mappers do a fairly good job, but still require a lot of work and tuning. Have you ever tried to update an object containing a child collection using Hibernate? It is not a straight forward operation.

what would be better?

Alternatives have cropped up over the years to fill this void. XML and object databases in the early part of the decade. “NoSQL” solutions more recently. I’ll describe what I would like to see in a “full featured” data store in today’s world and what systems come close to meeting those needs.

find-grained caching

Firstly, fine-grained caching or the ability to manage cache content; for example, keeping thousands of user logins cached but only the first fifteen blog posts. Ehcache in conjunction with Hibernate is a good example. In relatively complex applications this is important, you don’t want to fill memory with items that just aren’t going to get accessed that often.

searching

Next is searching for data. Searching gives us the ability to add facets and complete user generated lookups. Lucene is a great example of a full fledged searching product. I don’t think a data store needs to be quite as feature rich, but it seems like a natural way to find data.

flexible schema

Thirdly, a less rigid data definition system would also make sense. In other words, I’d like to be able to add or remove a column of data without restarting the system. Most relational systems can handle this, but are not well suited for adding columns on-the-fly at the application level.

versioning / transactions

Also, I’d also like to see an “immutable” or “versioning” approach to follow the current trend of non-locking in the multicore and functional programming worlds. Basically this means a new copy of the “tuple” (or at least the changed data) with each save, with “rollbacks” occurring during contention situations. This would allow the data to be stored and extracted against a timeline, much like version control software works today. This technique would also create a simple transaction model.

scaling

Another important consideration is scaling and redundancy are important. Servers are now commodities. Virtualization and the cloud permit massive scaling. Any new data storage solution has to take this mentality into account.

security

Finally, a more robust security model would be beneficial. Most relational databases will allow security at the database, table, and even column. Configuring such security with the application is much like adding or removing columns, it just isn’t feasible. Maybe creating separate “stores” with their own set of security credentials (i.e. a store for each user’s payment data) might reduce the potential damage an attacker could wield?

what else is out there?

Many things. There are a ton of systems, so I will attempt to categorize the most recent “NoSQL” solutions and how they fit (or don’t fit) into the basic needs described above.

the proposal

Not being able to find a perfect fit to the requirements listed above, I propose a data system that stores and indexes maps of data. A structure like this:

name: Mr. Data
title: Data Master
description: I am a block of data, please don't lose me or my changes


Would be found in this search:

description: bl*


or this search:

title: "Data Master"


and would produce a language specific construct like a java.util.Map or python dictionary. These maps would be packed into lists.

query system

Queries would be handled in a functional style with chained manipulations of data. For example:

query("occupation: student").sort().limit(5)


would query for records by occupation, sort the results, and limit the size of the results to five objects. This “chaining” would allow for customized extensions.

saving data

“Tuples” might be directories with metadata files. The directory name would be a unique identifier. So a record with four saves would look like this:

afm2acaa-asd2/
afm2acaa-asd2/metadata
afm2acaa-asd2/4
afm2acaa-asd2/2
afm2acaa-asd2/3
afm2acaa-asd2/1


The metadata file might contain the pointer to the latest save and a data version number:

latest: 4
version: 0.1


If two threads attempted to save new versions concurrently, they would produce temporary files:

afm2acaa-asd2/thread1
afm2acaa-asd2/thread2


The only synchronized code would be updating the “metadata” file. If “thread2″ finished after “thread1″, the “commit” operation would fail because it referenced an older version. The “thread1″ changes would be moved into place as:

afm2acaa-asd2/5


and the metadata file could be updated accordingly. This separate file based structure would allow for grepping and easier backups at the file system level. Unfortunately, there would be a large number of files and directories, which could dratamtically impact performance with big datasets.

caching

Caching could be optimized by additions to the query parameters. For example:


cache("five.students").query("occupation: student").sort().limit(5)


would cache the results of the query under the name “five.students”.

relationships

In theory, the results could be combined to produce “relationships”:

cache("five.students").combine(
"class", query("occupation: student").limit(5),
"name", query("type: class")
)


would inject the “class” map into the “student” where the “class” name matches the “student” class attribute. There would be no foreign keys, so data consistency would be up to the application.

replication

Replication is another thorny problem. I think a “shard manager” might be appropriate solution. It would need to run queries against all the database instances and combine the results. Or maybe be configured with some kind of “lookup” algorithm. This might also be a good location to handle caching. Of course, this approach would have to be transparent to the client and easily configured. And managers would need to be able to accept new stores on the fly. And we would want to be able to run multiple managers to avoid a single point of failure.

security and “stores”

Keeping identifying information for “tuples” could be handled with subdirectory “stores”. So our earlier example would look like this:

b2asco21-rjx2/
b2asco21-rjx2/metadata
b2asco21-rjx2/afm2acaa-asd2/metadata
b2asco21-rjx2/afm2acaa-asd2/4
b2asco21-rjx2/afm2acaa-asd2/2
b2asco21-rjx2/afm2acaa-asd2/3
b2asco21-rjx2/afm2acaa-asd2/1


The store metadata could contain a user, password combination:

user: test@test.com
password: 1234


or even an private key:

certificate: ...


A client would have to provide credentials before accessing the store data or caches.

what do you think?

Is this a boneheaded idea? Is there something already out there? Maybe this approach lacks features that would complete the “picture”? If this is actually interesting to you, please let me know. Maybe we can form a new project around it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

text text text 
 
keyword
 
text text text 
text text text 
text text text 
text text text 
text text text 
text text text 
text text text 
 
insert below
 
text text text 
text text text 
text text text 
text text text 
text text text 
text text text

and you needed to insert text from file “ins.txt” at the location “insert below”. All you need to do is run this series of commands:

1

find . -name "*.txt" -exec grep -l keyword {} \; | xargs sed -i '/insert below/r ins.txt'

Basically this command interprets as:

1. in the current directory, find all the files with extension “txt”
2. from those files, find only the ones containing the phrase “keyword”
2. insert the text from “ins.txt” below the phrase “insert below” in those matching files

Of course regular expressions work across the board for this type of operation, with both the “find” and “grep” portions.

For this project our needs were fairly basic: sensitive information and functionality needed to be isolated from our production applications, to safeguard from a potentially compromised server. Web Services wouldn’t typically be a first choice, considering options like REST or even a simple servlet, but integration with the standard runtime gave hope that they would be easier to manage and might provide some additional utilities for free.

The architecture is simple, a few JPA annotated beans managed by hibernate, utility classes, and a single service class to expose the functionality. We use Jetty for most of our deployments, seeing JBoss or even Tomcat as adding unnecessary complexity.

getting started

Adding the appropriate JSR-181 annotations was not difficult, the only “tweaking” we did was to manipulate the default namespace with the “targetNamespace” attribute on the “WebService” annotation. We also decorated the methods we wanted exposed as services, although that might not be a necessary step.

To begin deployment, we first used the jars from the GlassFish JAX-WS reference implementation. The “WSServlet” class is included, which can be configured in web.xml to handle incoming requests. For basic functionality, the reference implementation worked well. It would even generate a WSDL on the fly, reducing much maligned configuration hassles.

We had to add a “sun-jaxws.xml” file to our “WEB-INF” directory:

1
2
3
4
5

<?xml version="1.0" encoding="UTF-8"?>
<endpoints version="2.0" xmlns="http://java.sun.com/xml/ns/jax-ws/ri/runtime">
  <endpoint implementation="(full class name)" name="calculator" 
    url-pattern="/ws/Calculator"/>
</endpoints>

After firing up Jetty, we used poster to test the service, by POSTing a simple SOAP request similar to this:

1
2
3
4
5
6
7
8
9
10

<?xml version="1.0"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <soapenv:Body>
    <calulator:add xmlns:calculator="http://calculator.fake.org/">
      <first>1</first>
      <second>2</second>
    </calculator:add>
  </soapenv:Body>
</soapenv:Envelope>

Here is a portion of our web.xml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

<?xml version="1.0" encoding="UTF-8"?>
<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    xsi:schemaLocation="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd">
<display-name>ws</display-name>
  <listener>
    <listener-class>
      com.sun.xml.ws.transport.http.servlet.WSServletContextListener</listener-class>
  </listener>
  <servlet>
    <servlet-name>ws</servlet-name>
    <servlet-class>com.sun.xml.ws.transport.http.servlet.WSServlet</servlet-class>
    <load-on-startup>1</load-on-startup>
  </servlet>
  <servlet-mapping>
    <servlet-name>ws</servlet-name>
    <url-pattern>/ws/*</url-pattern>
  </servlet-mapping>
  <session-config>
    <session-timeout>30</session-timeout>
  </session-config>
</web-app>

adding security

When we moved beyond the basics, requiring authentication to secure our service, it quickly became apparent we would need to switch to the full metro stack. Metro version 1.4 operated differently in some places; for instance, any classes backed by interfaces had duplicate “mesage”, “portType”, and “binding” elements when using the “wsgen” utility.

We decided x509 “mutual certificate” security would meet our needs by both encrypting messages and requiring a trusted key for authentication. Unfortunately, there are no annotations to configure this service, the WSDL files would now have to be manually updated. This seems to defeat the original purpose of annotations, but was necessary to get the “free” WSIT security features. To use predefined WSDL files, we added “wsdl” attributes to the appropriate services in “sun-jaxws.xml”:

1
2
3
4
5

<?xml version="1.0" encoding="UTF-8"?>
<endpoints version="2.0" xmlns="http://java.sun.com/xml/ns/jax-ws/ri/runtime">
  <endpoint implementation="(full class name)" name="calculator" 
      url-pattern="/ws/Calculator" wsdl="WEB-INF/wsdl/CalculatorService.wsdl"/>
</endpoints>

We used Netbeans to generate a sample security configuration that was transplanted into an existing WSDL, following the steps in this link. The “wsp:Policy” element and it’s children at the bottom of the generated WSDL needed to be added, along with the correct namespace references from the top level “definitions” element. Here is an example of integrating the security references in the binding section of the document (see generated WSDL for examples). This:

1
2
3
4
5
6

<operation name="register">
  <soap:operation soapAction="" />
  <input>
    <soap:body use="literal" />
  </input>
<output>

will become:

1
2
3
4
5
6
7
8
9
10
11

<operation name="register">
  <soap:operation soapAction=""/>
  <input>
    <soap:body use="literal"/>
    <wsp:PolicyReference URI="#CalculatorPortBinding_Input_Policy"/>
  </input>
  <output>
    <soap:body use="literal"/>
    <wsp:PolicyReference URI="#CalculatorPortBinding_Output_Policy"/>
  </output>
</operation>

Netbeans can also be used to generate the client WSDL files (i.e. “Calculator.xml”) and “wsit-client.xml” configuration file. See here.

testing it out

Now we needed to generate keystores and client libraries to test the security configuration. Creating keystores for metro was confusing, apparently they had to be generated with “SubjectKeyIdentifier”s as described here. In a nutshell, we used this script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

#!/bin/sh
 
# create service keystore
rm *.p12 *.pem *.jks *.cer
openssl req -x509 -days 3650 -newkey rsa:1024 -keyout servicekey.pem \
    -out servicecert.pem -passout pass:changeit
openssl pkcs12 -export -inkey servicekey.pem -in servicecert.pem -out service.p12 \
    -name myservicekey -passin pass:changeit -passout pass:changeit
keytool -importkeystore -destkeystore servicestore.jks -deststorepass changeit \
    -srckeystore service.p12 -srcstorepass changeit -srcstoretype pkcs12
keytool -list -keystore servicestore.jks -storepass changeit | less
keytool -exportcert -alias myservicekey -storepass changeit 
    -keystore servicestore.jks -file service.cer
keytool -printcert -file service.cer | less
rm *.pem *.p12
 
# create client keystore
openssl req -x509 -days 3650 -newkey rsa:1024 -keyout clientkey.pem \
    -out clientcert.pem -passout pass:changeit
openssl pkcs12 -export -inkey clientkey.pem -in clientcert.pem -out client.p12 \
    -name myclientkey -passin pass:changeit -passout pass:changeit
keytool -importkeystore -destkeystore clientstore.jks -deststorepass changeit \
    -srckeystore client.p12 -srcstorepass changeit -srcstoretype pkcs12
keytool -list -keystore clientstore.jks -storepass changeit | less
keytool -exportcert -alias myclientkey -storepass changeit -keystore clientstore.jks \
     -file client.cer
keytool -printcert -file client.cer | less
rm *.pem *.p12
 
# add certificates to corresponding stores
keytool -import -noprompt -trustcacerts -alias myclientkey -file client.cer \
    -keystore servicestore.jks -storepass changeit
keytool -import -noprompt -trustcacerts -alias myservicekey -file service.cer \ 
    -keystore clientstore.jks -storepass changeit
rm *.cer

The resulting stores had references to their paired certificates, so attempts to connect with other client certificates would not work. We also had to create client code, using the following ant target:

1
2
3
4
5
6
7
8

<target name="ws.import" depends="build.etc.path">
  <taskdef name="wsimport" classpathref="build.classpath"
      classname="com.sun.tools.ws.ant.WsImport"/>
  <wsimport debug="true" destdir="${build.dir}/classes"
      wsdl="${ws.wsdl}" keep="false" package="${ws.package}">
    <produces dir="${build.dir}/classes/${ws.build.dir}" includes="*" />
  </wsimport>
</target>

making the client flexible

Finally, we wanted to configure the client to work in our QA and Live environments, using variable trust stores and service endpoints. The “wsp:Policy” element at the bottom of the client WSDL (a.k.a Calculator.xml):

1
2
3
4
5
6
7
8

<wsp:Policy wsu:Id="CalculatorPortBindingPolicy">
  <wsp:ExactlyOne>
    <wsp:All>
      <sc:KeyStore .../>
      <sc:TrustStore .../>
    </wsp:All>
  </wsp:ExactlyOne>
</wsp:Policy>

can be modified to look for a properties file:

1
2
3
4
5

<wsp:Policy wsu:Id="CalculatorPortBindingPolicy">
  <wsp:ExactlyOne>
    <wsp:All><!-- handled by client-security-env.properties --></wsp:All>
  </wsp:ExactlyOne>
</wsp:Policy>

A file “client-security-env.properties” was required in the classpath, that had similar entries:

1
2
3
4
5
6
7

keystore.url=etc/clientstore.jks
keystore.type=JKS
keystore.password=changeit
my.alias=myclientkey
truststore.url=etc/clientstore.jks
truststore.type=JKS
truststore.password=changeit

You can also modify the endpoint URL during the instantiation of service with code similar to:

1
2
3
4

java.net.URL url = new URL("http://localhost:8080/ws/calculator");
javax.xml.namespace.QName qname = 
    new QName("http://calculator.fake.org/", "CalculatorService");
Calculator calc = new CalculatorService(url, qname).getCalculatorPort();

additional options

We were getting some strange stack traces during the client configuration from to references to a “MEX” service. “MEX” was enabled by adding this line:

1
2
3

<endpoint name="sts_mex" url-pattern="/ws/calculator/mex"
  implementation="com.sun.xml.ws.mex.server.MEXEndpoint" 
  binding="http://www.w3.org/2003/05/soap/bindings/HTTP/"/>

to sun-jaxws.xml. Unfortunately, we still got stack traces and had to download and modify “com/sun/xml/ws/mex/client/MetadataClient.java”. There was a “suffixes” String array that need to have it’s elements switched from { “”, “mex” } to { “mex” , “” }. We then packaged the resulting class with our client code.

For peace of mind, we also used Wireshark to verify that the SOAP messages were actually being encrypted.