Switching from DEVONthink to Obsidian

Love changes, a thug changes, and best friends become strangers … .
Nas

I have written a lot of postings about DEVONthink. It was and still is a great app. In essence, it’s a kind of macOS Finder «replacement». An app in which you can manage your files. It detects duplicated files based on file content not file name, allows you to quickly find files, and has a plethora of other functions. See, for example, these postings:

But while I compared it to the Enterprise rising out of the clouds of a gas giant regarding how powerful it is and how easy you can underestimate it, it also became the wrong tool for me. It became kinda like the Star Trek reboot.

In a sense, my relationship with the software became poisoned with the DEVONthink 3 release. They removed the Three Panes View and were rather … well, they not only ignored complaints. Not only mine. And the user «support» went on a kind of confrontation course. So, imagine you try to convince a company that a removed functionality is crucial to your workflow and the customer «service» argues with you that the software is better without it.

Not the way you want to deal with customers for whom knowledge management is a core job task.

After that change, yeah, DEVONthink wasn’t the same. Not only because my workflow became less efficient, but also due to the reaction. Even a talk with a company representative did not change that. Very disappointing, considering the customer support so far was excellent. I once asked how to do something and I got a script back which did the task. Yeah, waow. And then you have someone who argues with the customers and is unreceptive to serious concerns.

Luckily, DEVONthink is not a data island — you can easily export your data. Just drag the group from DEVONthink into a folder outside of DEVONthink. As DEVONthink can have files with the same name in the same group, some renaming might occur. And check the Log for errors. But yeah, DEVONthink allows you to leave with your data (otherwise I would not have used it in the first place).

So, when a colleague recommended Obsidian, I did take a look. In a sense, Obsidian is what I tried to do during the times when I used DokuWiki. As a wiki comes with a lot of typing overhead, I once wrote a «Ferret Frame» that used JavaScript and PHP to make working with the wiki easier. It opened the DokuWiki in a Browser frame and allowed you to quickly change content (incl. using Templates), create pages, etc. It worked, but later I changed to Circus Ponies Notebook and then (when it folded) to DEVONthink.

Obsidian combines the best of what DokuWiki ever provided (enhanced by the Ferret Frame) with the quickness and ease of use of … well, hard to find a comparison.

Update: Thinking about it, DEVONthink was more about handling files, while Obsidian is more about notes. Showing information and allowing for quick edits. Sure, you can search for rtf files in DEVONthink and create them with a click on an icon, but you still need to move the mouse a lot and switch between mouse and keyboard. In Obsidian, you can simply use the cmd + o menu to find or create notes. The focus in different and much more in line with the way I work.

So, yeah, no more DEVONthink for file and data management. All files are now exported and available in Finder folders. After all, search can be limited in the search filed to filenames (use name:) or kind of file (use kind:) in addition to doing a full content search. Mostly I am looking for images (kind:image) or specific file names (e.g., articles, e.g., name:meyer) anyway.

I have also created Inboxes — folders that are added to Finders sidebar (just drag the folder there), for example, a Collection Inbox, one folder for as inbox for images, etc. No difference to using a quick add to DEVONthink. Perhaps even better, considering that DEVONthink was not really geared towards so-called power-users. I do not think the software was tested with GBs of data (e.g., whole libraries worth of PDF files).

Backup is also faster, given that no database has to be backed up. Only downside, if I did drag a file from DEVONthink to the Desktop, it did copy the file. Now, if I want to do the same from a Finder folder, I have to remember to press the option key.

As for the main part of DEVONthink, the notes in rtf documents. That was surprisingly easy. I used pandoc (called from R, which I also used find all .rtf files and tidy the filesnames). Pandoc is able to convert files, so (in R) via:

system(paste0("pandoc ", cleanSource, " -f rtf -t markdown_mmd -s -o ", cleanTarget))

it is possible to convert lots and lots of files (was part of a for loop that went through all found rtf files in a DEVONthink export folder and created a similar folder structure with markdowns instead of rtf). The few rtfd files (which contain images) were harder to deal with (had to do it manually). Still, the files are not in Obsidian yet (only a tiny amount). I want to add them into the next structure (mostly via copy-pasting part of the content). But at least they are no longer rtf files. (I have added the code I think I used below. ABSOLUTELY NO WARRANTY.)

So yeah, so far, so good. Here’s hoping that Obsidian does not go the way of the Dodo, or of Circus Ponies Notebook.

And as for DEVONthink — is there still a use for it (for me)? Yeah, I still like the Website-to-PDF plugins (but am switching to Print => Save as PDF). I also continue to use it as an oversized RSS Reader. And it is still useful to find duplicated images (it can do so with indexed files, i.e. files that are not copied into a DEVONthink database). But that’s pretty much it. It was fun until the loss of the Three Panes View, which cost the software my loyalty. Took a while to find something else, but, yeah, found it.

Looking back, that was a long way from DokuWiki to Circus Ponies to OmniOutliner (could not handle the amount of data) to DEVONthink back to the Wiki-like solution of Obsidian. But it was a spiral upwards, almost every step of the way (only OmniOutliner was an express elevator to hell).

And — as usual — I see in the renewed productivity that it works.

R Code to use pandoc (has to be installed separately, IIRC) to convert rtf files to markdown. Done very … not-computer-science-like. I had to recreate some lines and ABSOLUTELY NO WARRANTY. But it might serve as a start. But seriously, better go for the principle and use a programming language you like (or better code in R). This is very very hacky (and not in a good way).

# Find RTF Files ----

allRTFFiles <- list.files("RTFConversion/NotesDT", pattern = ".rtf", recursive = TRUE)

# sort (might be needed as the directories have to created hiearchically)
allRTFFiles <- sort(allRTFFiles)

dataTable <- tibble(filename = allRTFFiles) %>% rownames_to_column(var = "id")

cleanName <- function(x) {
    x <- str_replace_all(x, "\\'", "")
    x <- str_replace_all(x, "\\(", "")
    x <- str_replace_all(x, "\\)", "")
    x <- str_replace_all(x, "\\,", "")
    return(x)
}

for(i in 1:length(allRTFFiles)) {
    if( !str_detect(allRTFFiles[[i]], "\\.rtf") ) { next }
    sourcePathFilename <- paste0("RTFConversion/NotesDT/", allRTFFiles[[i]])
    targetPathFilename <- paste0("RTFConversion/NotesMD/", allRTFFiles[[i]])
    targetPathFilename <- str_replace(targetPathFilename, ".rtf", ".md")
    
    # create directory (have to do it sequentially)
    targetPath <- str_split(targetPathFilename, "/", simplify = TRUE)
    targetPath <- paste0(targetPath[1:length(targetPath)-1], collapse = "/")
    ifelse(!dir.exists(cleanName(targetPath)),
           dir.create(cleanName(targetPath), recursive = TRUE),
           ("Folder exists already"))
    
    # convert rtf to markdown
    cleanSource <- str_replace_all(sourcePathFilename, " ", "\\\\ ")
    cleanSource <- str_replace_all(cleanSource, "\\'", "\\\\'")
    cleanSource <- str_replace_all(cleanSource, "\\(", "\\\\(")
    cleanSource <- str_replace_all(cleanSource, "\\)", "\\\\)")
    cleanSource <- str_replace_all(cleanSource, "\\,", "\\\\,")
    cleanTarget <- str_replace_all(targetPathFilename, " ", "\\\\ ")
    cleanTarget <- str_replace_all(cleanTarget, "\\'", "")
    cleanTarget <- str_replace_all(cleanTarget, "\\(", "")
    cleanTarget <- str_replace_all(cleanTarget, "\\)", "")
    cleanTarget <- str_replace_all(cleanTarget, "\\,", "")
    system(paste0("pandoc ", cleanSource, " -f rtf -t markdown_mmd -s -o ", cleanTarget))
    targetPathFilename <- str_replace_all(targetPathFilename, "\\'", "")
    targetPathFilename <- str_replace_all(targetPathFilename, "\\(", "")
    targetPathFilename <- str_replace_all(targetPathFilename, "\\)", "")
    targetPathFilename <- str_replace_all(targetPathFilename, "\\,", "")
    
    # get the folder names for tagging ----
    path <- str_replace_all(str_split(allRTFFiles[[i]], "/", simplify = TRUE), " ", "_")
    pathTags <- paste0("#", path[1:length(path)-1], collapse = ", ")
    
    # remove tick marks and remove unnecessary stuff ----
    changeFile <- read_lines(targetPathFilename)
    changeFile <- str_replace(changeFile, "author: Daniel Wessel", "")
    changeFile <- str_replace_all(changeFile, "`", "")
    first <- 0
    last <- 0
    for(i in 1:length(changeFile)) {
        if((changeFile[[i]] != "") & changeFile[[i]] != "  ") {
            if(first != 0) { last <- i } else { first <- i }
        }
    }
    changeFile <- changeFile[first:last]
    
    changeFileTags <- c(changeFile, " ", pathTags)

    # save
    write_lines(changeFileTags, targetPathFilename)
    
    print(targetPathFilename)
}

ORGANIZING CREATIVITY

How to generate, capture, and collect ideas to realize creative projects.

Switching from DEVONthink to Obsidian