I wonder if the real problem isn't that a lot of corporate engineering teams are becoming less literate then they were in the age of typewriters and overnight mail.
Big problems have been tackled by large geographically dispersed teams dependent entirely on written communications in the pre-internet past so for today project teams to depend solely on being able to talk face-to-face to every member of the team at a regular basis might be more of a new problem then an old problem exposed by the digital age.
It could be that search and tagging is not an actual replacement for the archivist specialization and that we in order to relearn the art of running large project/organizations need to reintroduce dedicated archivist and technical writers to the process.
It could also be an consequence of how people are taught as maybe the practice of small fast projects lead to people and organisations who never really learn how to record ideas, debates and information.
It is part of the "agile" culture -- by which I mean the popular bastardization created by incompetent managers and incompetent developers, where anything resembling strategic thinking is dismissed as "waterfall" i.e. something the cool kids would never touch.
From the incompetent manager's perspective: Typing code is productive work, everything else (such as writing documentation) is not. I am paying you for implementing features for the customer, not writing notes for yourself and your team.
From the incompetent developer's perspective: Hey, my idiosyncratic code is self-documenting, and if people do not understand something, it's because they are stupid. Also, more job security is good for me.
Corporate programming is typically bent towards feature delivery and managing patches. Only feature delivery requires managing a project, but it's not new development scale. It involves one or two engineers at most, and anything that requires greater resources than that are looked at skeptically. It's my opinion that this puts us heavily out of practice.
Instead, I'd like to engage in work that is outside of our core business. I want to build our own dev tools at times, I'd like to run our own infra rather than toss it at Amazon, Google, or Microsoft and point the finger when it breaks. Having to work outside of a product we know inside and out, domains we're already very familiar with, will improve residual skills like documentation. The repetition of doing and the variety in what we do makes skilled and well-rounded engineering teams imo.
Speed is the issue. If IT is a cost center, the business doesn't see the documentation and don't care about it. They want the next feature. At least that's how it is at my company.
If the work-from-home thing goes wrong, this will be why.
A huge amount of institutional memory is tacit knowledge, things that don't go in an explicit document. Your corporate Wiki is great for concrete things like where the different logo files are, but you'll never be able to rebuild the business from the documents alone, even if they're very good.
Those idle moments when you roll over to a colleague's desk, or when there's a crisis and you all huddle together, that's when you learn a thing or two about how the sausage is actually made.
With work-from-home it's not impossible to make up for this, but it does require the business to understand that finishing the documents is not finishing the work.
Teams with bad documentation and which rely on tacit understanding are hugely vulnerable to people leaving the company and that knowledge walking out of the door, or people dying.
I’ve seen it myself multiple times - I’ve been hired to work on software with no or little documentation, where someone has retired or left, and there’s pretty much no choice but to start reimplementing parts of it because nobody understands any of it. It might be salvageable in some areas but when it’s my area (scientific/engineering code), lack of documentation kills projects, because nobody can work backwards easily and find out the assumptions which were made.
Which, unfortunately, is ignored quite often from people who haven't experienced "death" yet. If someone walks out the door, there is most likely a chance you can get in contact with them if you need a crucial information afterwards (depends, sure, but you get the gist) - with dead people, the only thing you can rely on is a spiritual medium.
I usually introduce it to management as "the bus factor" - how many people in this team would it take to be hit by a bus for the team to be completely fucked.
I'm currently working (and soon leaving) a job like this - there's someone that keeps so many things to themselves that if they were hit by a bus, nobody would be able to deploy anything, etc. which is just unacceptable IMO.
Shouldn’t this legitimately be in the domain of management? Documenting, building a framework for understanding how work is accomplished, interfaces with other departments, etc.?
In which case each manager is documenting their own process and knowledge.
When this person leaves they take their notes with them?
I don't know what teams you work on, but in the teams I've been based in, management have had a shaky grasp on technical aspects and wouldn't understand the documentation (if any) if it was put in front of them.
People (almost) always underestimate legibility problems. The terms used (eg knowledge problems, documentation, etc) betray this.
Where and when we can "solve" knowledge problems via codification, that work is now a commodity. It has already been outsourced, automated or relegated to organisational backwaters and low level employees.
>>many of the knowledge problems are cultural
I agree, but think the author is understimating both the problem and the "cultural" solution.
To codify a function is to change it entirely, like going from skilled craft production to injection molding and factory assembly. Everything changes, both the process and the output.
I think we're very limited in our capacity to intentionally affect culture. At best we can introduce surface level elements and hope that they permeate somewhat. The cultural problem is a lot like the knowledge problem, in this sense.
Comes down to individual vs. collective (organization) interests.
Think about what you're asking an employee to do: "spend a bunch of extra time articulating and internally publishing your knowledge so that you're more replaceable as an employee".
In an organization with low cohesion, this is a non-starter, no matter how shiny your "knowledge management" tools are. Job security is king in these environments.
If you have a knowledge silo problem, you probably have a deep management/HR problem.
Tools are never the solution, they support your solution. (Founded a VC financed startup 20+ years ago to index the implicit knowledge in heads, the startup failed but my Wiki engine was used by Atlassian for some years in their products :-)
I'm biaised because I work on a tool for this exact problem. Even if I agree with the article and the title, and I don't think it makes much sense.
There is no way to solve those "problems" they are not even problems, we can only be better at handling them. And even measuring this 'better' is not easy. But there is no solution to sharing information. If you can use a better tool, then why not use it ?
With all the saas tools now, more and more information is written somewhere. But we need a new paradigm and stop trying to make people document things on the same wiki. For example there is no better place for most code documentation than to be in the source code, why not let developers document their file there and create a 'links' to this information ?
The challenge is that there are two directions for the knowledge retrieval: top-down and bottom-up. The "problem" here is that failure to learn from past, well-characterized solutions means we will be forced to revisit problems experimentally (expensive) that could be solved by examining the documentation (relatively cheap). Literate programming and inline documentation is great for understanding the bottom-up picture, because you colocate the components that are needed for understanding. For top-down, you are often summarizing several projects or proposals and trying to explain trajectory and architecture, and necessarily will not always have an obvious "best" location for retrieval.
For example, consider an assembly line in a factory. Having work instructions and debugging manuals next to a station is great, but where should we place the information explaining why this station must be configured in particular way to prevent an issue six steps downstream? Do both stations now need to have that information documented? Do we have higher-level documentation that needs to be synchronized? How would we find information about the changes made while debugging that process which ultimately did not make the cut, but would be useful to know as negative experiments?
I would be curious to learn more about the tool you work on. I have never found a good solution for architecting layered information for retrievablility, only least-worst hacks.
Yes, having documentation close to source code is bottom-up and being able to index this documentation somewhere is the top-down. Some documentation are useful to have in both ways and some are not. But if you force people to use a new tool to store their documentation, some will stop documenting and some will start, the best solution is to allow both ways but to index it in a central place without having to duplicate information.
> I have never found a good solution for architecting layered information for retrievablility, only least-worst hacks.
I don't know if we will end up in a 'good solution' situation, as that's quite a complex topic, but we try to approach the problem with different ideas in mind :
- A great user experience, how to encourage people to document and search in the documentation (I always joke that, a documentation platform is good only when all salespeople of a company use it everyday)
- Spreading the ownership of content organization, how we can rely on 1-2% of employees having 'expert' responsibilities to create the right framework to collaborate in his group
- Letting anyone curate information even on content they are not responsible of. As people that like documenting or care about it are not the same than the ones having expert knowledge on the topic.
At the end it's always difficult to discuss theory on those kind of topics as the only thing that counts is that it works in practice. And that's also why tools are important, they are able to hide the theory and let users use simple things
Have you heard of pol.is or vtaiwan? Do you have opinions on them or of what they are trying to accomplish that you wouldn't mind sharing? There are others I'm having trouble remembering at the moment.
Letting anyone curate information even on content they are not responsible of. As people that like documenting or care about it are not the same than the ones having expert knowledge on the topic.
This is how a system should be for good faith contributors. It has the eternal wikipedia-style problem of truthiness always being in question though. In normal internal documentation, I don't imagine this being too much of a problem. It would be nice if good versions of these communication platforms, seemingly like dokkument, became available for more of the public at large.
I never heard about them, pol.is is interesting, I will read about it. As for vtaiwan I don't think there are similar problem that we are trying to tackle. It feels like vtaiwan is more like a forum where people who are willing to contribute can contribute, in an organization one big challenge is to onboard people who don't care about collaborating.
> It has the eternal wikipedia-style problem of truthiness
Wikipedia is the reference when talking about curation and collaboration, and being able to have an internal Wikipedia (not just the tool, but also the contributors, the readers, the overall system) is the goal. In a company you can create an approval system with experts accepting new changes that fix the issue about truthiness.
> It would be nice if good versions [...] became available for more of the public at large.
That's my personal long term goal, whether it's good practice, searching for a flat, a good restaurant, ... there are ways that crowd-curation can solve a lot of the issue we have today, there was an interesting article about it a few days ago on HN : https://news.ycombinator.com/item?id=29130017
One area that is a "knowledge wild west" is managerial and executive decision making. Apparently, people are expected to figure out how to handle an incalculable amount of business situations through osmosis or intuition.
My move into management roles quickly led me to plateau. I could only realistically improve my coaching skills. Decision-making skill seemed to be dependent on situational exposure, which meant upper management levels were only achievable after years of experience. This explains management demographics and the perception of the ambiguity of management roles. "What do managers do all day??"
I made a breakthrough though. I learned about Decision Modeling and Notation, which I earnestly embraced. I began modeling all sorts of situations I encountered as a manager. My ultimate failure was in not being able to apply it to the knowledge of people further up the totem pole. It's hard to explain to people that management can be largely systematized, especially when that itself is a paradigm shift that could be a threat to their own status.
It almost sounds like a version of 2nd law of thermodynamics: You cannot extract more organized knowledge (analogous to work) without discarding some unorganized information (analogous to heat).
The last mile problem here is the jump to people's brains and it isn't easily solved by technical means. It goes beyond UX or UI and is really an empathy issue.
Big problems have been tackled by large geographically dispersed teams dependent entirely on written communications in the pre-internet past so for today project teams to depend solely on being able to talk face-to-face to every member of the team at a regular basis might be more of a new problem then an old problem exposed by the digital age.
It could be that search and tagging is not an actual replacement for the archivist specialization and that we in order to relearn the art of running large project/organizations need to reintroduce dedicated archivist and technical writers to the process.
It could also be an consequence of how people are taught as maybe the practice of small fast projects lead to people and organisations who never really learn how to record ideas, debates and information.