Search and Replacement Techniques
From time to time it’s a good idea to turn back to the basics. Even after years of Emacs usage you will potentially find things you overlooked before - or new ways established without you noticing.
This week I searched for workflows to find and replace matches with Emacs. In this post I will summarize the different techniques I found and hopefully provide you with some new methods you might want to try or incorporate into your own workflow. I will contrast "the old way" with the "modern way" in each section, but notice that by modern I don’t necessarily mean better.
"Old" just means that the method has been built-in for a long time and "modern" means, external packages or newer additions to stock Emacs are used to achieve the task. As Emacs is big and the community even bigger the described methods are not near complete. Feel free to add you own suggestions in the comments.
TL;DR
My current favorites are:
-
objed to replace the symbol at point in current defun or buffer
-
anzu to replace a regex in current buffer
-
project (built-in) to replace a regex or symbol in a project
-
swiper and counsel, deadgrep to search and narrow down matches in a buffer or project
In buffer search and replace
The old way (isearch, query and replace, occur and keyboard-macros)
If you are new to Emacs and want to learn more about isearch
I
highly recommend
this
nice tutorial by Bastien Guerry.
To start a classic search and replace session in Emacs jump back to the beginning of the buffer with M-< and start a query replace with M-% or C-M-% (regex version) or the non-interactive versions M-x replace-string and M-x replace-regexp.
The most useful keys inside a query replace sessions are SPC to replace and move to next match, n to skip a match, ! to replace all remaining matches in the currrent file, Y to replace all matches in all upcoming files, N to skip the rest of matches in current the file.
If you ever want to repeat a replace command with slightly different input try C-x ESC ESC to get the last minibuffer command as an Elisp expression, although using the minibuffer history using M-p or M-n is usually enough.
Another neat thing about query replace in Emacs, which you will not
find in other Editors, is that you are allowed to use Elisp on the
matched groups of the regex. For example you could use this to replace
matches for "movie" with "film" and matches for "movies" with "films"
using the search term movie\(s\)? and \,(if \1 "films"
"film") as the replacement expression. Another common use
case is to transform numbers in the matches using the format
function.
If you are in the middle of a search and want to replace the used
search term in the whole buffer, there is one more built-in method I
would like to describe here. Namely you can invoke occur
(with your
current search term) using M-s o. This will give you a complete
listing of the matched lines. Now jump to the first match and start
recording a keyboard macro with F3 or C-x (, after editing
the match to your liking jump to the next match with M-g n.
Finally stop the recording of the macro with F4 or C-x ).
This will allow you to execute the macro with C-x e or F4. To execute until you reach the end of the buffer. You can use a zero prefix argument C-0 C-x e to repeat your edits on all the matches found.
Update: As hmelman points out you can use e in occur buffers to enter occur-edit-mode and after editing commit the changes with C-c C-c. This even works for multi-occur!
One final built-in technique that deserves a mention here, is that you can always restrict your replacements by narrowing your buffer before you start using one of the methods above. For example mark a paragraph with M-h and narrow to its region with C-x n n. To exit the narrowing use C-x n w to widen the buffer to its old size again.
The modern way (anzu, swiper, helm, wgrep, mc, iedit, symbol-overlay)
The "old way" using isearch
can be improved by using the
anzu package. It shows you the
regex matches and regex groups highlighted in the buffer. Another
package that does this is
visual-regexp. The
replacements are shown inline, too. This is very helpful for
constructing the right search and replacement terms. Another neat
feature of anzu is that it shows you the number of matches in the mode
line.
Many people use packages which show interactive previews of the search matches like swiper or helm. For both of these you can switch to an editable buffer of the matches. Using helm-swoop you can press C-c C-e to switch to edit mode and apply changes to original buffer by C-x C-s. For swiper you press C-c C-o to switch to the ivy-occur buffer. There you press C-x C-q to edit it using wgrep and C-x C-s to commit your changes.
The way you edit the matches is totally up to you. You can start a simple query replace session as described in the last section or go fancy using multiple-cursors, iedit or symbol-overlay. Those three packages can be used to quickly replace the symbol at point, too. If that’s all you want to do, using one of these in the first place is a bit faster and more convenient for this task.
My way
I like to use swiper to get an
overview of matches, but every now and then isearch
is handy, too.
For example I like to use it for replacing matches in the ivy-occur
buffer. To quickly navigate and replace symbols in the current buffer
or defun I use my own package called
objed and for replacing a regexp I
use anzu.
Search and replace across multiple files or buffers
There are different ways to solve this task. Either you collect a list of files first and then run a query replace on them or you immediately collect matches in files of interest and edit the matches afterwards. Depending on the used method the set of files can be specified by a glob pattern, a regex, a TAGS file or special project files (.gitignore, .project).
The old way
Find files/matches
To narrow to the set of files you are interested in you have several options:
-
M-x find-name-dired will prompt you for a root directory and a filename pattern, the results are collected in a dired buffer
-
M-x dired will prompt you for the directory. Mark all the files with t or to narrow to specific files, first insert sub directories using C-u i (add "-R" to recurse) and mark files matching a regex using % m
-
Create a TAGS file for your project. For Elisp you can generate one using M-! find . -name '*.el' -print | etags -. To append other files for example add C files to the table use find . -name '*.[ch]' -print | etags --append -. Many other languages are supported see M-x man etags.
To collect a list of matches in a set of files you can use one of the following:
-
M-x rgrep will collect lines matching a glob pattern in matched files in a *grep* buffer.
-
M-x vc-git-grep will collect lines matching a glob pattern for files in your git project and respects your .gitignore.
Replace matches
-
If you went the M-x dired route use Q and you will be prompted for query/substitution afterwards. This will start a regular query replace session which will run through all marked files.
-
If you created a TAGS file you can now use M-x tags-query-replace to run query replace on all tags in your TAGS file.
-
When you used M-x rgrep or M-x vc-git-grep you could define a keyboard macro like described above.
The modern way (swiper, wgrep, helm, projectile, project-find-regexp)
M-x rgrep, vc-git-grep and the various counsel commands like counsel-rg/ag/pt/git-grep can be used in conjunction with wgrep the same way as described in the previous section.
For helm there are similar methods using M-x helm-projectile-grep, helm-occur or helm-do-grep.
Using projectile and projectile-replace, projectile-replace-regexp you can run an interactive query replace on project files, too. It uses tags-query-replace under the hood.
With Emacs 25 comes a new built in project library. Using project-find-regexp you can search in the current project which recognizes various version control types automatically. This pops up a buffer where you can navigate the matches with n, p and start an interactive query replace on the matches with r.
My way
To replace matches I prefer project-find-regexp which is convenient to use and built-in.
For interactive file search and grep I use swiper and counsel-rg again and sometimes the neat deadgrep package.
Closing words
The methods described often don’t save the changed files. To save the changes you made to all buffers you can either use C-x s followed by ! or call ibuffer and mark all unsaved buffers with * u followed by S to save them. Using ibuffer has the benefit that you can clean up by closing all marked files afterwards with D.