Emacs: Find files anywhere

From Amit’s Thoughts

Update: [2017-12-29] See the update for 2017

A typical way to use Emacs is to open lots of files as buffers. In the default Emacs setup, the command to switch to a file depends on whether the file is open:

I don't want to have to remember whether a file is open. Instead, I want the command to depend on whether I'm switching to a file or a non-file. I also want to find files without having to switch folders first, because I work on lots of small projects in different folders. I use these bindings:

In a previous post, I described my previous attempt, using helm-for-files to open files from many different directories, using locate (mdfind on Mac). As part of that, I improved my Mac OS mdfind setup to include all my text files. I was hoping that I could make mdfind fast and precise enough that I'd use it all the time. Unfortunately I couldn't get the queries to run faster than 300 milliseconds, and it didn't feel fast enough to run on every keystroke. There are other emacs packages to do this but I ended up with my own custom setup to make this:

I took inventory of the files I wanted to open: they have to be text, they are in my home directory, and they are not in one of a set of subdirectories (such as .git or ~/Library/). I have only a few thousand of these. So why not just load them all into emacs? It'll make the search-as-I-type really fast.

After playing around with various options, I ended up with the idea of extending the notion of "recent files" to include all of the ones I might open. This will cover most everything. I can then fill in the rest with locate/mdfind. To do this, I dynamically bind recentf-list before calling helm-for-files.

This setup is specific to my system; I hope it gives you ideas for making your own.

The initial setup has an Emacs component and a shell component.

(defvar amitp/global-file-list '() "List of most files I work with")

(defun amitp/helm-for-files ()
  "Global filename match, over all files I typically open"
  (interactive)
  (let ((helm-ff-transformer-show-only-basename nil)
        (recentf-list (mapcar 'abbreviate-file-name amitp/global-file-list)))
  (helm-other-buffer
   '(helm-source-recentf)
   "*helm for files*")))

(define-key global-map [(hyper t)] 'amitp/helm-for-files)

To fill amitp/global-file-list I run a shell script in cron:

# build-filename-list.sh
mdfind -onlyin $HOME "kMDItemContentTypeTree = 'public.text' || kMDItemFSName = 'Makefile'" \
   >~/.global-file-list.txt

(and also add other filename extensions that aren't classified as 'public.text') Then I read it into Emacs:

(defun read-file-into-lines (filename)
  "Read file, split into lines, return a list"
  (with-temp-buffer
    (insert-file-contents filename)
    (split-string (buffer-substring-no-properties (point-min) (point-max)) "\n" t)))

(setq amitp/global-file-list (read-file-into-lines "~/.global-file-list.txt"))

This is fast. And it does what I want. But the order of results isn't ideal. If I could sort them better, the most likely results would show up first, and I'd have to type less to get to the file I want. And I also wanted to make sure I was able to find files that weren't captured in the precomputed list.

I decided to sort in this way:

1. open buffers first
2. current directory's files
3. recently opened files
4. the global file list
5. anything mdfind can come up with (helm-source-locate)
6. any other file system path (helm-source-find-files)

(defun amitp/helm-for-files ()
  "Global filename match, over all files I typically open"
  (interactive)
  (let ((helm-ff-transformer-show-only-basename nil)
        (recentf-list
         (append (amitp/buffer-file-names)
                 (helm-skip-boring-files (directory-files default-directory t))
                 recentf-list
                 amitp/global-file-list)))
    (helm-other-buffer
     '(helm-source-recentf helm-source-locate helm-source-find-files)
     "*helm for files*")))

(defun amitp/buffer-file-names ()
  "A list of filenames for the current buffers"
  (loop for filename in (mapcar 'buffer-file-name (buffer-list))
        when filename
        collect filename))

It was better, but not quite enough. The output of mdfind isn't sorted, but I can roughly sort it by putting it into groups:

1. files I've modified in the past 1 day
2. files I've modified in the past 2-4 days
3. files I've modified in the past 5-14 days
4. files I've modified in the past 15-45 days
5. all other files

(mdfind -onlyin $HOME "kMDItemContentTypeTree = 'public.text' && kMDItemFSContentChangeDate > \$time.today(-1)";\
 mdfind -onlyin $HOME "kMDItemContentTypeTree = 'public.text' && kMDItemFSContentChangeDate > \$time.today(-4) && kMDItemFSContentChangeDate <= \$time.today(-1)";\
 mdfind -onlyin $HOME "kMDItemContentTypeTree = 'public.text' && kMDItemFSContentChangeDate > \$time.today(-14) && kMDItemFSContentChangeDate <= \$time.today(-4)";\
 mdfind -onlyin $HOME "kMDItemContentTypeTree = 'public.text' && kMDItemFSContentChangeDate > \$time.today(-45) && kMDItemFSContentChangeDate <= \$time.today(-14)";\
 mdfind -onlyin $HOME "kMDItemContentTypeTree = 'public.text' && kMDItemFSContentChangeDate <= \$time.today(-45)"\
) >~/.global-file-list.txt

(the real script also includes all the other filename extensions that aren't classified as public.text) Definitely better. Combined with recentf, this means that the "working set" of recent files is rather short, and I don't have to type much to get to the file I want.

I've needed other heuristics as well (some posted in comments), and I continue to refine those. I can now use Cmd-T somewhat like I do in a browser, where I open a new tab and then start typing in a url or query. Another cute trick: in browsers, Cmd-Shift-T opens a recently closed tab; I do something similar in Emacs:

(define-key global-map [(hyper shift t)] 'helm-recentf)

Overall, I'm reasonably happy with this system, but I'm still looking for ways to make it better. Check out different ways to find files globally and see if anything fits your own needs.

Update: [2014-07-13] I wanted it to run even faster. I now start with helm-source-recentf, which is fast, and then add helm-source-locate and helm-source-find-files only on a keypress. One of the things I realized is that helm-recentf lets me type words out of order, whereas helm-locate does not, so it's useful to keep the two separate.

(defun amitp/helm-for-files ()
  "Global filename match, over all files I typically open"
  (interactive)
  (let ((helm-ff-transformer-show-only-basename nil)
        (recentf-list
         (mapcar 'abbreviate-file-name
                 (append (amitp/buffer-file-names)
                         (helm-skip-boring-files (directory-files default-directory t))
                         recentf-list
                         amitp/global-file-list))))
    (helm
     :sources '(helm-source-recentf)
     :buffer "*helm for files*")))

(defun amitp/helm-locate (candidate)
  "Fallback when helm-recentf doesn't find what I want"
  (interactive)
  (helm :sources '(helm-source-locate helm-source-find-files)
        :buffer "*helm locate*"
        :input helm-input
        :resume 'noresume))

(defun amitp/helm-for-files-fallback ()
  (interactive)
  (helm-quit-and-execute-action 'amitp/helm-locate))

(bind-key "C-l" 'amitp/helm-for-files-fallback helm-generic-files-map)

Update: [2014-09-29] It'd probably be cleaner to use helm-filesets but I haven't yet changed my configuration to use that.

Labels: emacs