When Data Becomes Code


If you’ve hung out on #emacs for a while, chances are you’ve been recommended ibuffer for advanced buffer management (like, killing many buffers at once). There is a horror lurking beneath its pretty interface though and it starts out with a customizable:

(defcustom ibuffer-always-compile-formats (featurep 'bytecomp)
  "If non-nil, then use the byte-compiler to optimize `ibuffer-formats'.
This will increase the redisplay speed, at the cost of loading the
elisp byte-compiler."
  :type 'boolean
  :group 'ibuffer)


(defun ibuffer-compile-make-eliding-form (strvar elide from-end-p)
  (let ((ellipsis (propertize ibuffer-eliding-string 'font-lock-face 'bold)))
    (if (or elide (with-no-warnings ibuffer-elide-long-columns))
        `(if (> strlen 5)
             ,(if from-end-p
                  ;; FIXME: this should probably also be using
                  ;; `truncate-string-to-width' (Bug#24972)
                  `(concat ,ellipsis
                           (substring ,strvar
                                      (string-width ibuffer-eliding-string)))
                   ,strvar (- strlen (string-width ,ellipsis)) nil ?.)

(defun ibuffer-compile-make-substring-form (strvar maxvar from-end-p)
  (if from-end-p
      ;; FIXME: not sure if this case is correct (Bug#24972)
      `(truncate-string-to-width str strlen (- strlen ,maxvar) nil ?\s)
    `(truncate-string-to-width ,strvar ,maxvar nil ?\s)))

(defun ibuffer-compile-make-format-form (strvar widthform alignment)
  (let* ((left `(make-string tmp2 ?\s))
         (right `(make-string (- tmp1 tmp2) ?\s)))
       (setq tmp1 ,widthform
             tmp2 (/ tmp1 2))
       ,(pcase alignment
          (:right `(concat ,left ,right ,strvar))
          (:center `(concat ,left ,strvar ,right))
          (:left `(concat ,strvar ,left ,right))
          (_ (error "Invalid alignment %s" alignment))))))

(defun ibuffer-compile-format (format)
  (let ((result nil)
        ;; We use these variables to keep track of which variables
        ;; inside the generated function we need to bind, since
        ;; binding variables in Emacs takes time.
        (vars-used ()))
    (dolist (form format)
       ;; Generate a form based on a particular format entry, like
       ;; " ", mark, or (mode 16 16 :right).
       (if (stringp form)
           ;; It's a string; all we need to do is insert it.
           `(insert ,form)
         (let* ((form (ibuffer-expand-format-entry form))
                (sym (nth 0 form))
                (min (nth 1 form))
                (max (nth 2 form))
                (align (nth 3 form))
                (elide (nth 4 form)))
           (let* ((from-end-p (when (cl-minusp min)
                                (setq min (- min))
                  (letbindings nil)
                  (outforms nil)
                  min-used max-used strlen-used)
             (when (or (not (integerp min)) (>= min 0))
               ;; This is a complex case; they want it limited to a
               ;; minimum size.
               (setq min-used t)
               (setq strlen-used t)
               (setq vars-used '(str strlen tmp1 tmp2))
               ;; Generate code to limit the string to a minimum size.
               (setq minform `(progn
                                (setq str
                                        `(- ,(if (integerp min)
             (when (or (not (integerp max)) (> max 0))
               (setq max-used t)
               (cl-pushnew 'str vars-used)
               ;; Generate code to limit the string to a maximum size.
               (setq maxform `(progn
                                (setq str
                                        (if (integerp max)
                                (setq strlen (string-width str))
                                (setq str
                                        'str elide from-end-p)))))
             ;; Now, put these forms together with the rest of the code.
             (let ((callform
                    ;; Is this an "inline" column?  This means we have
                    ;; to get the code from the
                    ;; `ibuffer-inline-columns' alist and insert it
                    ;; into our generated code.  Otherwise, we just
                    ;; generate a call to the column function.
                    (ibuffer-aif (assq sym ibuffer-inline-columns)
                        (nth 1 it)
                      `(,sym buffer mark)))
                   ;; You're not expected to understand this.  Hell, I
                   ;; don't even understand it, and I wrote it five
                   ;; minutes ago.
                    (if (get sym 'ibuffer-column-summarizer)
                        ;; I really, really wish Emacs Lisp had closures.
                        ;; FIXME: Elisp does have them now.
                        (lambda (arg sym)
                            (let ((ret ,arg))
                              (put ',sym 'ibuffer-column-summary
                                   (cons ret (get ',sym
                      (lambda (arg _sym)
                        `(insert ,arg))))
                   (mincompform `(< strlen ,(if (integerp min)
                   (maxcompform `(> strlen ,(if (integerp max)
               (if (or min-used max-used)
                   ;; The complex case, where we have to limit the
                   ;; form to a maximum or minimum size.
                     (when (and min-used (not (integerp min)))
                       (push `(min ,min) letbindings))
                     (when (and max-used (not (integerp max)))
                       (push `(max ,max) letbindings))
                      (if (and min-used max-used)
                          `(if ,mincompform
                             (if ,maxcompform
                        (if min-used
                            `(when ,mincompform
                          `(when ,maxcompform
                     (push `(setq str ,callform
                                  ,@(when strlen-used
                                      `(strlen (string-width str))))
                     (setq outforms
                           (append outforms
                                   (list (funcall insertgenfn 'str sym)))))
                 ;; The simple case; just insert the string.
                 (push (funcall insertgenfn callform sym) outforms))
               ;; Finally, return a `let' form which binds the
               ;; variables in `letbindings', and contains all the
               ;; code in `outforms'.
               `(let ,letbindings
    ;; We don't want to unconditionally load the byte-compiler.
    (funcall (if (or ibuffer-always-compile-formats
                     (featurep 'bytecomp))
             ;; Here, we actually create a lambda form which
             ;; inserts all the generated forms for each entry
             ;; in the format string.
             `(lambda (buffer mark)
                (let ,vars-used
                  ,@(nreverse result))))))

(defun ibuffer-recompile-formats ()
  "Recompile `ibuffer-formats'."
  (setq ibuffer-compiled-formats
        (mapcar #'ibuffer-compile-format ibuffer-formats))
  (when (boundp 'ibuffer-filter-format-alist)
    (setq ibuffer-compiled-filter-formats
          (mapcar (lambda (entry)
                    (cons (car entry)
                          (mapcar (lambda (formats)
                                    (mapcar #'ibuffer-compile-format formats))
                                  (cdr entry))))

(defun ibuffer-clear-summary-columns (format)
  (dolist (form format)
    (when (and (consp form)
               (get (car form) 'ibuffer-column-summarizer))
      (put (car form) 'ibuffer-column-summary nil))))

(defun ibuffer-check-formats ()
  (when (null ibuffer-formats)
    (error "No formats!"))
  (let ((ext-loaded (featurep 'ibuf-ext)))
    (when (or (null ibuffer-compiled-formats)
              (null ibuffer-cached-formats)
              (not (eq ibuffer-cached-formats ibuffer-formats))
              (null ibuffer-cached-eliding-string)
              (not (equal ibuffer-cached-eliding-string ibuffer-eliding-string))
              (eql 0 ibuffer-cached-elide-long-columns)
              (not (eql ibuffer-cached-elide-long-columns
                        (with-no-warnings ibuffer-elide-long-columns)))
              (and ext-loaded
                   (not (eq ibuffer-cached-filter-formats
                   (and ibuffer-filter-format-alist
                        (null ibuffer-compiled-filter-formats))))
      (message "Formats have changed, recompiling...")
      (setq ibuffer-cached-formats ibuffer-formats
            ibuffer-cached-eliding-string ibuffer-eliding-string
            ibuffer-cached-elide-long-columns (with-no-warnings ibuffer-elide-long-columns))
      (when ext-loaded
        (setq ibuffer-cached-filter-formats ibuffer-filter-format-alist))
      (message "Formats have changed, recompiling...done"))))

Another weird one is that the extracted autoloads for ibuffer-ext.el reside in ibuffer.el, but that’s the lesser evil of the two.

Credits go to holomorph for discovering that maintenance nightmare.

PSA: Emacs is not a proper GTK application


Update: I’ve been pointed to an emacs-devel discussion about giving Emacs a proper GTK frontend, most comparable to the Win32 and NS frontends (which are free from X11isms). This would be a better approach than what’s been outlined below, with the Cairo code being repurposed for the drawing bits.

Daniel Colascione’s excellent write-up on bringing double-buffered rendering to Emacs has prompted me to do the same on a set of questions that can be occasionally spotted on #emacs:

  • Which GUI build of Emacs shall I choose?
  • What’s the difference between the GTK build and the other builds of Emacs on Linux?
  • Does the GTK build run on Wayland?
  • Does the GTK build run on Broadway?
  • Why does Emacs not render as nicely as <insert more modern text editor>?

If you’ve ever programmed a GUI application with a modern/popular GUI toolkit, you’ll have noticed that while it gives you loads of desirable features, it forces you to structure your application around its idea of an event loop. In other words, your application is forced to react asynchronously to user events. Games are pretty much the only major kind of graphical application that can get away with doing their own event loop, but end up doing their own GUI instead.

Now, the issue with Emacs is that it does its own event loop, pretending that the frontend is a textual or graphical terminal. It’s pretty much the graphical equivalent of a REPL and at the time it only had a text terminal frontend, this way of doing things worked out fairly well. However, by the time users demanded having pretty GTK widgets in Emacs, it became clear that more involved hacks were needed to make that happen. This is why Emacs runs the GTK event loop one iteration at a time, pushes its own user events into it (to make widgets react) and a plethora of more hacks to reconcile their rendering with everything done by X11.

In other words, Emacs is more of a X11 application plus your favorite widgets. The choice of GUI toolkit to build it with is mostly irrelevant, save an infamous bug with the GTK3 frontend that can crash the daemon. Emacs will therefore not run on a pure Wayland system or under Broadway in the browser. If anyone would want to make that happen, either the GTK frontend would need to yank out everything X11 (unlikely as it’s deeply entrenched) or to create a new frontend doing platform-agnostic drawing (the Cairo feature being a prime candidate for that).

Further reading material:

Honesty is the Best Policy

;;; Please do not try to understand this code unless you have a VERY
;;; good reason to do so.  I gave up trying to figure it out well
;;; enough to explain it, long ago.

This precedes paredit-forward-sexps-to-kill which appears to be code that deals with the problem that commands operating on S-expressions may fail mysteriously if there’s no trailing newline afterwards.

Credits go to contrapunctus.

Brutal Workarounds


Today I took another stab at my abandoned Emacs Lisp IRC bot and thought to myself that it would be nice if it were able to notify me on RSS and Atom feed updates. Now, I’m not terribly fond of reinventing the wheel, so like every good programmer I dared taking a look at existing solutions, like News Ticker:

(defun newsticker--do-xml-workarounds ()
  "Fix all issues which `xml-parse-region' could be choking on."

  ;; a very very dirty workaround to overcome the
  ;; problems with the newest (20030621) xml.el:
  ;; remove all unnecessary whitespace
  (goto-char (point-min))
  (while (re-search-forward ">[ \t\r\n]+<" nil t)
    (replace-match "><" nil t))
  ;; and another brutal workaround (20031105)!  For some
  ;; reason the xml parser does not like the colon in the
  ;; doctype name "rdf:RDF"
  (goto-char (point-min))
  (if (re-search-forward "<!DOCTYPE[ \t\n]+rdf:RDF" nil t)
      (replace-match "<!DOCTYPE rdfColonRDF" nil t))
  ;; finally.... ~##^°!!!!!
  (goto-char (point-min))
  (while (search-forward "\r\n" nil t)
    (replace-match "\n" nil t))
  ;; still more brutal workarounds (20040309)!  The xml
  ;; parser does not like doctype rss
  (goto-char (point-min))
  (if (re-search-forward "<!DOCTYPE[ \t\n]+rss[ \t\n]*>" nil t)
      (replace-match "" nil t))
  ;; And another one (20050618)! (Fixed in GNU Emacs
  ;; Remove comments to avoid this xml-parsing bug:
  ;; "XML files can have only one toplevel tag"
  (goto-char (point-min))
  (while (search-forward "<!--" nil t)
    (let ((start (match-beginning 0)))
      (unless (search-forward "-->" nil t)
        (error "Can't find end of comment"))
      (delete-region start (point))))
  ;; And another one (20050702)! If description is HTML
  ;; encoded and starts with a `<', wrap the whole
  ;; description in a CDATA expression.  This happened for
  ;; http://www.thefreedictionary.com/_/WoD/rss.aspx?type=quote
  (goto-char (point-min))
  (while (re-search-forward
          "<description>\\(<img.*?\\)</description>" nil t)
     "<description><![CDATA[ \\1 ]]></description>"))
  ;; And another one (20051123)! XML parser does not
  ;; like this: <yweather:location city="Frankfurt/Main"
  ;; region="" country="GM" />
  ;; try to "fix" empty attributes
  ;; This happened for
  ;; http://xml.weather.yahoo.com/forecastrss?p=GMXX0040&u=f
  (goto-char (point-min))
  (while (re-search-forward "\\(<[^>]*\\)=\"\"" nil t)
    (replace-match "\\1=\" \""))
  (set-buffer-modified-p nil))

I guess I’ll not be using this and go for a special-purpose solution instead without over a decade old workarounds. After all, I’m not stuck in 2005 with Emacs 21…

Close Enough


byte-opt.el is one of those files Jamie Zawinski laid his golden hands on. It seems that back in the days, there wasn’t much of a concern about Emacs Lisp execution speed until he got annoyed enough to bolt on an optimizer. Its sources start with a wonderful quote:

“No matter how hard you try, you can’t make a racehorse out of a pig. You can, however, make a faster pig.”

I recommend reading it to get an idea what compiler jargon like “peephole optimizer” could possibly mean. During my last study, I found this curious piece of code:

(defun byte-optimize-approx-equal (x y)
  (<= (* (abs (- x y)) 100) (abs (+ x y))))

So, according to this 99 and 100 are equal. Awesome!