With all the political and vendor hype surrounding technological change, it was refreshing to read a recent Economist article “Of plumbing and promises” analyzing the factors behind the great recession of 2008. There was apparently an enormous backlog of unconfirmed trades, leaving buyers of credit default swaps (insurance policies for financial derivative products) without proof of purchase. According to the article, “Everyone knew that it was a paper mess but no one wanted to sign up to a single standard.”

The infrastructure weakness was compounded by a behavioral one: a failure to understand the risks inherent in various products before it was too late.

As I read the article, I couldn’t help thinking how innovations in knowledge work are subject to the same two factors — an infrastructure failure (lack of good metadata and an information quality assurance system) and a failure of imagination (the belief in the efficacy of computer software as opposed to human judgment).

One way to minimize the pitfalls is to focus on increasing work group productivity for a specific business process: e.g. sales, competitor intelligence, public relations. For more information, see the feature article in the April 2012 issue of the Montague Institute Review, “SharePoint search tools for workflow support.”

Alon makes the following points:

1. Choosing a problem is less an act of production (i.e. a quantitative measure of the number of high quality papers published) but rather an act of “nurturing” (i.e. qualitative measure of learning and self-motivation).

2. The ideal problem is characterized by both feasibility and high interest. In other words, it’s better to by-pass the easy task (which is feasible but not very important) and focus on an issue with a large potential impact on lower costs, higher revenue, or long-term competitiveness.

3. Problems should be matched to a professional’s experience. For example, a new employee should be guided toward easy problems, while an experienced senior manager might select a “grand challenge” problem that can be subdivided into many good, smaller projects.

4. Do not rush the problem-definition stage. Alon has a rule for new students and postdocs: do not commit to a problem before 3 months have elapsed. How often do we fail to allocate enough time to studying an issue and then have to write the whole effort off as a failure?

5. To choose a good problem, we need to reflect our own world view. Are we motivated by visual aesthetics or by abstract ideas? Supporting dogma or undermining commonly held truths? Preferring techniques or logical proofs? Since many good problems require the collaboration of multidisciplinary teams, it’s also necessary to help people understand the assumptions, tools, and techniques of other knowledge specialties.

6. Allow for failure, diversions, and confusion. Problem solving rarely follows the initial project plan. In the midst of confusion can come a new take on the issue or an entirely new, more important problem. Sailing into the unknown again and again takes courage. How well do we reward it?

Although written for the laboratory scientist, the article has implications for other kinds of professionals, since in our global knowledge-based economy the realms of academia, government, and business overlap and merge. I would even go so far as to say that choosing the right problem is even more important than finding the right “solution,” especially as you move up the organizational hierarchy.

Related articles

• Swets launches a Mendeley special edition (scienceintelligence.wordpress.com)

1. Technology leads, humans follow. I’ve worked with a couple of visionaries who understood both the potential for new technologies AND the possible applications for them, but most of us simply react to new products and features as they appear. The process of digesting new technology is a time-consuming process of evaluation, budget approval, implementation, and training. The result is that technology diffusion in large organizations can take a long time. The effect is likely to be incremental because new features are measured against existing business process. In fact, an organization’s capabilities become disabilities in the face of disruptive growth opportunities (see “Information services for corporate growth,” a review of Clayton Christensen’s book, The Innovator’s Solution). For that reason — and because intangibles like customization and training are hard to estimate — return on investment is often sub-optimal (and may even be negative).

2. Technology pursues monopoly. In our economic system, the way to cash in on new technology (or the data that it generates) is to maneuver into monopoly position. Think of railroads, Microsoft, Google, and Amazon. Benefits include economies of scale (Google), workflow integration (Microsoft), a large selection and low costs (Amazon). But when monopolies disappear (and they always do),disruption can be massive, especially for large organizations. Nobody likes to hear about the dangers and costs of obsolescence, conversion, and migration, but they are always lurking somewhere in the future (see “Review: Does IT matter?“).

3. Technology unwinds business models. Everybody knows this, but it can take a long time to happen, and most managers have a hard time coping with the implications for their industry and organization. I think the reason is that entrenched industries in our economy system are based on mastery of technologies, the production of physical artifacts, and the business ecologies that have evolved to monetize them. As soon as the Web browser appeared, it was obvious that the impact on the education and publishing industries would be profound. Yet, here we are nearly 20 years later still agonizing over an artifact called a “book” and its implications for bookstores, readers, libraries and three technology giants — Amazon, Apple, and Google (see “Publish or perish“).

When I did a SCIP workshop on the use of the Internet in competitive intelligence (CI) in the early 1990′s, Kodak had a highly regarded CI function. There was apparently no lack of information about the switch to digital cameras, even as early as 1979. But, while its Japanese competitor Fujifilm is thriving, Kodak filed for bankruptcy this month (see “The Last Kodak Moment?”).

4. Technology empowers end users. I’ll never forget the thrill of producing a nearly professional newsletter with the Apple Laserwriter, creating a taxonomy and A – Z Index with Filemaker, or checking email on my iPhone. Cloud-based services like Lulu.com and WordPress are even easier to use and require less up-front investment. End-user development can make organizations more agile by enabling employees to capitalize on opportunities without over-burdening the IT staff, but it has broader strategic implications (see “End-user developers: A critical corporate asset“). From a career perspective, it means distancing ourselves from roles based on things (i.e. computers, books, libraries) and focusing instead on developing our intellectual capital. From an organizational perspective, it means more emphasis on extranets (see “SharePoint use in extranets“). From society’s perspective, it means substituting mentoring for financial capital (see “The way companies are getting financed is completely changing“).

Related articles

• End-user development tools vs. black box software (montagueinstitute.wordpress.com)
• What Kodak teaches us about disruptive innovation (reportr.net)
• What Could Kill Apple, Google Or Microsoft? (forbes.com)

Shortly thereafter, I read a profile of Silicon Valley libertarian billionaire Peter Thiel and an article about a related group of people who design the most sophisticated technology on the planet but fear its dark potential.

These men live in a rarefied world where their intelligence opens doors and attracts large sums of money. I do not begrudge them either their smarts, opportunities, or wealth. After all, these are the kind of people who drive the innovation that allows people like me to have fun and make a living. But I wonder whether we are doing them (not to mention society) a disservice by failing to expose them early on to the kind of experience that the rest of us get in the school of hard knocks.

Some of these smart men are part of a subculture of thinkers, entrepreneurs, and nongovernmental organizations raising money to help defend the human race from its own creations. They include Saving Humanity from Homo Sapiens, the Lifeboat Foundation, and the Singularity Institute for Artificial Intelligence (of which Thiel is a backer). These may be worthy efforts, but how effective will they be? I’m skeptical, when I read that Thiel is known for his extreme views on taxes (there shouldn’t be any), women’s suffrage (it never should have happened), and his backing of an initiative to create a floating, visa-free “entrepreneurship and technology incubator” off the coast of San Francisco.

I believe in thinking outside of the box, but maybe smart young men would benefit from reading Leo Tolstoy as a counterbalance to Ayn Rand and competing to invent something cheap and useful for the bottom of the pyramid (the 99%).

The big exception to my Luddite tendencies is technology that allows me to be more creative. That explains why I spent almost $5,000 for the first Apple LaserWriter in 1986, started building Filemaker applications when the current version was 2 (it’s now 11), and am fascinated with new end-user development tools.

According some analysts, by 2014 end-user developers may constitute 25% of users in large organizations. Maybe when this happens, people like me will be able to tap a robust support system to help us identify and evaluate new tools and integrate new applications. For what these services might look like, see “End-user developers: A critical corporate asset.”

As much as I love the creativity, speed, and control of end-user development tools, I’m increasingly uneasy about “black box” automation. In “When algorithms control the world,” Jane Wakefield of the BBC describes a transatlantic cable between Canada and the UK built primarily to serve the needs of algorithmic traders, who will be able to send shares from London to New York and back in 60 milliseconds. She quotes Kevin Slavin, an algorithm expert, who says:

“We are running through the United States with dynamite and rock saws so an algorithm can close the deal three microseconds faster, all for a communications system that no humans will ever see.”

In “The Next Economic Revolution,” Alex Planes of The Motley Fool, questions the conventional wisdom that neither education nor entrepreneurship can solve the unemployment in a consumption-based economy. He says:

“Accepting a world in which human labor is largely supplanted by technological solutions requires us to radically re-imagine the foundations of our economy, which is a step most are not prepared to take.”

The geeky side of me can’t wait for next breakthrough in end-user development tools while the Luddite side is cautious about technology that promotes unsustainable growth or creates market instability.

Related articles

• The Personal Computer Is Dead (technologyreview.com)

Both Amazon and Google are creating ecosystems (not just new software features) that leverage the creative work of individuals to generate transaction or advertising revenue. In Amazon’s case, this means giving authors:

• direct access to data about book sales in individual markets;
• one-on-one communication between authors and readers that used to happen only on book tours;
• professional editorial and graphic design assistance.

The catch is that authors do not receive a cash advance, but their works are exposed in Amazon’s hugely popular electronic marketplace with nifty features like Search Inside This Book.

With YouTube’s new service, content creators can:

• search for legally re-usable videos from such organizations as C-CSPAN, Public.Resource.org, Voice of America, Al Jazeera, and others;
• mark their own videos with the Creative Commons CC-BY license so that others can use and remix their work.

Contrast those creator-friendly features with objectives from the Microsoft governance document (italics are mine):

• “… processes are understood and lines of communication follow predictable paths;
• business usage translates to business value via known pathways and is measurable;
• business users know how to use the available features as intended;
• usage growth is predictable and managed.”

From the IT point of view, predictability is good, maybe even necessary. But creative people are most valuable when they develop unconventional solutions, pursue out-of-network relationships, and use tools in new ways. True, creative effort in a business environment does need to be measured. The marketplace determines the metric for Amazon and Google, but for intranets, the yardstick is not so clear.

For “back room” corporate operations where efficiency and low cost are key objectives, meaningful metrics are fairly straightforward. For customer-facing functions, such as sales and product development, measurement of the sort described in the governance document is notorious for being inaccurate, irrelevant, or both.

Should large business and government organizations try to implement creator-friendly ecosystems like those offered by Amazon and Google? If so, how?

Image via Wikipedia

The IT world borrowed the taxonomy concept from biology to compensate for the fact that users didn’t know what to type in the search box. It was hard for them to figure out what a Web site was about and zero in on their area of interest. The taxonomy (a hierarchical list of topics) served as a Web site table of contents and, in some cases, the equivalent of a back-of-the-book topic index. Eventually, it was also used to make search engines “smarter” by letting users narrow search results by topic and display related information, such as a list of experts.

As we add bottom-up and inside-out relationships to the prevailing top-down model of content creation and search, taxonomies look less useful and harder to manage. What will replace them? A recent post on the Freebase forum by Paul Houle of Ontology² gives us a glimpse of what’s in store when we publish and subscribe to metadata in a semantic web format.

“One of my favorite oddities in Freebase is the case of ‘J. Edgar Hoover’, who, according to the auto-suggest,  has the primary type of “Film Actor.”  I guess that’s nominally true because he appears as himself in at least two films.

When I first saw his record,  he was typed as a “Politician” but there wasn’t any documentation of that — no fact about what office he’d held.  I felt uncomfortable with that typing because I thought a “Politician” was somebody who’d been elected or who ran for office, but looking at the docs I saw it could apply to an appointee as well.  I added a CVT to state that he was director of the FBI, but this still doesn’t communicate the importance of his position.

There are a number of ways to evaluate the “importance” of a topic by looking at relationships.  For instance,  if a topic has a lot of quads,  it means that somebody thought the topic was worth documenting.  You could also evaluate a movie by counting the rewards it has received or know that an actor is a :FamousActor if they’ve got a star on the Hollywood Walk of Fame. All these scoring systems are imperfect, but faced with 22 million topics, you need to have some sense of priorities.

J. Edgar Hoover, one of the most powerful figures in U.S. politics in the 20th century, gets shortchanged by the simple scoring system because he’s not well documented. Call him a bureaucrat, call him a cop, call him a tyrant, Freebase doesn’t have a vocabulary to describe his career, so he gets consistently beat by the likes of :Elton_John unless you put some strong opinions in your T-box [schema].”

There are, of course, several good programs on the market that integrate metadata repositories with SharePoint, but my Filemaker database already does the thesaurus part (as well as many other things). I just needed the Filemaker/SharePoint connector.

Remember that you can import Excel-created csv files into SharePoint 2010, but SharePoint doesn’t have true thesaurus relationships. The advantage of using Filemaker is that you can create and customize databases with little or no help from a programmer. For more on this topic, see:

Integrating taxonomies with SharePoint BCS

How to create SharePoint taxonomies

Customizing & governing the SharePoint search system

“The not-native-to-either-party language is like a neutral territory, a halfway point where the joys and difficulties of communication are shared equally. I speak more slowly, knowing that my listeners may not ‘hear’ me like a native; they return the favor. We all leave out tricky turns of phrase.”

This kind of communication is essential in knowledge base publishing, even when all communication is in English. I think we tend to use the “phrase book” when communicating because it’s easier — you know, use terms like “Web 2.0″ or “taxonomy.” One problem is that the words can mean different things to different people. A bigger problem is that the term doesn’t convey the scope of the underlying ecosystem to the non-native speaker. The result is often poor decision-making and projects that fail to live up to their potential.

See also “Using plain language to build a brand“

Maybe I’m part of a trend.  In “Education needs a digital-age upgrade,” the author discusses the work of Cathy Davidson, co-director of the annual MacArthur Foundation Digital Media and Learning Competitions. In a new book, Now You See It, Davidson  says that she was appalled when her students at Duke, who produced witty and incisive blogs for their peers, turned in disgraceful, unpublishable term papers. After studying the matter, she concluded, “Online blogs directed at peers exhibit fewer typographical and factual errors, less plagiarism, and generally better, more elegant and persuasive prose than classroom assignments by the same writers.”

The same issue of the New York Times featured another article on the same theme. In When knowledge isn’t written, does it still count, the author discusses the idea that much has been lost to Wikipedia because of stickling rules of citation and verification. One example is dabba kali, a children’s game played in the Kerala state of India. Someone created a Wikipedia article in the local language, Malayalam, that included photos, a drawing and a detailed description of the rules, but no sources to back up what was written. Other than, of course, the 40 million people who played it as children. According to current rules, the article would have been deleted from the English Wikipedia edition.

Does this mean we shouldn’t package knowledge in traditional vehicles like books and articles? Of course not. But those of us who learn, write, and teach for living need to experiment with new formats. Then the question becomes: when should we use one format instead of another, and what’s the most efficient data structure to store the raw material?

See also Packaging knowledge: the evolution of the book from the Montague Institute Review.