Tag: work-in-progress

Posted on
by

I’m interested in building up a personal store of knowledge – good ideas, data, memories, and notes – and saving it in a way that I can quickly and easily make use of this information. Without some external way of saving things, I tend to forget them. The dream is that these ideas and pieces of information don’t have to be re-thought up or re-found each time but can be explored to rapidly generate more ideas. Ideally this would help me solve problems and help generate new knowledge for the wider society.

In reality this has turned out to be much harder than I expected. In my experience, it is easy to save information but hard to find it and make use of it later. The vast majority of my files have never been used after I saved them. When I do happen to remember a piece of information that I want to look up, I tend to have trouble finding it. This unfortunate situation surprised me. I had bought into the idea that there is essentially no cost to saving things digitally so I should save everything, and second that digital tools should make it easy to find things. I was wrong on both counts.

The difficulty of storing information became a real problem for me in grad school when I was doing research for my PhD in physics. I had to deal with countless journal articles, presentations, calculations, brainstorming sketches, experimental data, notes from talks/classes, and different versions of code (I wasn’t using version control, aaahhh!). At first, I thought my problems were just caused by the wrong organizational system. So I tried different systems, but despite having advantages and disadvantages they didn’t solve the underlying problem for me.

Thanks to my misadventures, I’m learning that forgetting is a feature not a bug. When I try to remember all my ideas and information, the default organizational systems struggle and while specialized tools can help they also tend to silo information. It turns out, there is a need to limit the inflow of information and the amount of work in progress. I need the information and ideas that are relevant to me, and everything else is unlikely to be needed, and should be let go. In a way my organization system for saving ideas and information is an extension of my brain. My brain is quite good at forgetting things (sometimes too good!). So while it is important to have good tools and organization, shouldn’t my system also forget things?

The default ways of organizing tend to be digital analogues to the way we store papers: papers on the desk, in notebooks, and in folders. These systems fail in similar ways to paper based systems. The simplest approach is to just put everything you need on the desktop, but this quickly fails if you have more than around ten files. The next simplest is to create notebooks; essentially storing the files in chronological order (for example by starting the filename with the date). This helps because you can quickly find a file if you remember when you worked on it, which tends to happen for things you recently worked on. However this approach makes it hard to see connections between ideas from different times1. You can’t re-arrange the ideas as you learn more; they are fixed in chronological order.

The next strategy that I’ve tried is putting files in a folder structure according to topic. This seems like a reasonable idea, but quickly becomes quicksand for ideas. Inevitably, I need to change the folder structure as I learn to better reflect the association between my files. The changes make it hard to remember where things are and breaks any references to file paths. As I create more folders, I find it difficult to remember the context underlying the relationships between folders. This can lead to folders with mashed up collections of files and sub-folders that don’t make sense.

Based on these struggles, I have turned to more specialized tools as a way to solve my organizational challenges. They can make a huge difference, but can create silos of information. For example, I started using Zotero in grad school, to keep track of all my research articles. This was a huge improvement over putting papers in topic based folders and trying to cram paper keywords into the filenames. Zotero has useful tools that allowed me to generate bibliographies automatically and to save papers directly from the browser. But now I had a separate store of papers that wasn’t tightly integrated with my research data and information; it was in a separate silo. This turned into a problem when I needed to reinstall my operating system. All my other files were restored but I had trouble recovering my Zotero papers. Eventually I was able to recover them, but it highlights that separate silos increase the complexity of securing my data.

Ultimately tools won’t help if the information is not useful. Marie Kondo says throw away all papers (except what you really need)2, and I think the same could be said for digital information too. She says this because according to her core principle, your things should “spark joy” and she finds papers rarely spark joy. Do my digital files and information spark joy? The majority probably don’t. I’m not totally sure because most are in these old folders that I never look at.

Information tends to bring me more joy when it is connected to something bigger, and this is where tools can help. For example, the “Untitled” file in an old folder called “New” is probably not useful. The context is lost; it is just a fragment. In contrast, a note in an Evernote notebook is part of something bigger. Evernote allows the notes to be displayed in different ways which can help the context to be recovered, and it is easy to search. It’s also easier to make connections between different notes with tags or with links.

Based on these experiences, I have begun to seriously fight the accumulation of unnecessary fragments and now try to only keep things that are part of a “whole”. One practice that has helped a lot is to channel the inflow of information. By default, there many different places that new files are saved, which is bad because it’s hard to evaluate what should be kept. To make things more manageable, I create a single “Inbox” folder where any new file is saved temporarily until it can be saved in an appropriate place (I got the idea from this blog post3). This little change gave visibility to the problem, because I could easily see files building up. Now that the files were in one place, it was more straightforward to decide what to keep and what to delete (although I still put it off sometimes!). This is a small step toward protecting my digital space and being intentional about what I allow in.

Limiting the amount of work in progress is also critical. Work in progress can be many things: unfinished projects, ideas that need to be pursued, or emails that need to be responded to. Too much work in progress is bad for multiple reasons (Agile frameworks like Kanban4 and Scrum5 are dedicated to limiting work in progress), but in regard to storing information, the unfinished work means that there are lots of associated files and information that don’t bring joy. Instead of being sources of useful information, these incomplete files require additional work and are difficult to understand because of missing context. One tactic that helps limit the work in progress is to create a single “Current Projects” folder for things that are being actively worked on (this is another good idea from this blog post3). With all the active work in one place it is easier to see when things are getting overwhelming. When that happens, I need to let go of things that I am never going to do and work to finish the projects that are important.

Organization requires deep change in behavior. I am part of the knowledge organization system, so if the system is going to succeed then I need to be working too. This means being more intentional with what I keep and with how I keep the things that are important. The hard part is letting go. Sometimes that means deleting now and sometimes it means letting something be buried in the bottom of a list of notes. Either way this reserves the prime spots for the most useful information. As part of my system, I also need to focus on what is most important. This is a huge challenge because it requires saying “No” to some demands, and risking the disapproval of others. When those demands come from bosses, significant others, or children it’s difficult to say no. But by setting limits, I will be able to make the most of my limited resources and give the most back to those in my life.

The effort is worth it for me. As I have been implementing these changes, I have found that I spend less time searching for files, notes or projects from a few months ago. It still happens sometimes, but usually when I didn’t follow my own rules! My store of information is not some magically useful thing that I will use in the future, but a limited resource that I rely on and enjoy now. It has room to grow, with plenty of unneeded information that can be pruned and lots more that can be added in an integrated way. This practice is helping me to think more clearly. Letting go of information, forgetting, is helping me to make connections between the important ideas that are left.

  1. David B. Clear, “Zettelkasten — How One German Scholar Was So Freakishly Productive”, https://writingcooperative.com/zettelkasten-how-one-german-scholar-was-so-freakishly-productive-997e4e0ca125.
  2. Kondo, Marie. the life-changing magic of tidying up. 2014 p 96.
  3. Mark Virtue, “Zen and the Art of File and Folder Organization”, https://www.howtogeek.com/howto/15677/zen-and-the-art-of-file-and-folder-organization/.
  4. David J. Anderson, Kanban: Successful Evolutionary Change for your Technology Business.
  5. Kenneth S. Rubin, Essential Scrum: A Practical Guide to the Most Popular Agile Process.

Posted on
by

When developing software, I want to build applications quickly. In my sometimes painful experience, low quality work gets in the way of this goal. There are a lot of problems with poor code quality, but I want to focus today on how code quality is directly tied to the amount of work in progress. Low quality code leads to large work in progress which in turn causes slow and uncertain delivery time of new features.

I tend to think of low quality code as any code that leads to defects that require rework, i.e. bugs. There is an inherent challenge in this definition because it’s hard to know what will cause a bug until the bug shows up. But from experience, I know there tend to be more bugs when I write the code quickly, working alone, and without writing or running tests. There are a lot of different ways to write bad code, but those are some of the common ways I do it : ).

The low quality code really starts to be a problem when I “finish” some features and move on to developing new features. The poorly written code is not really done. Most likely it still needs some rework to really function the way it is intended. So in reality, I have more work on my plate than I can handle efficiently. Major bugs crop up more frequently interrupting me and causing the time to finish any feature to increase.

It turns out that there is no way to get around the fact that more work in progress leads to longer delivery time; it’s Little’s Law. If you have a system with items coming in that have to wait in a queue before they can exit the system, then the number of items in the queue is directly proportional to how long, on average, that it takes to process a single item. This makes sense. Long lines mean long wait times. But when it comes to planning work, it is easy to think that I can get everything done at once.

To achieve a quicker cycle time, I need to insist on high quality work to avoid the pile of work in progress from unexpected bugs. A faster feedback cycle is super beneficial. With complex products each bit of work that we complete gives us new insight into the system, and ideally we would like to incorporate those learnings into deciding what to do next. The only way we can do that effectively is to have short delivery time of features.

This is my understanding of the connection between work in progress and delivery time related to my experience, but I learned about this connection first in Kanban: Successful Evolutionary Change for your Technology Business by David J Anderson and more recently in Essential Scrum: A Practical Guide to the Most Popular Agile Process by Kenneth S. Rubin.

I hope to write upcoming posts about specific things that I have found helpful in my efforts to improve the quality of my code.