9.3 Building with Bookdown, GitHub, and Zotero

by Jack Dougherty and Ilya Ilyankou

We are building this open-access book with open-source tools—namely Bookdown, GitHub, and Zotero—and openly sharing the knowledge we acquire so that others may improve on the process. We welcome feedback from digitally-minded authors and software developers on Twitter or directly on the GitHub repo for this page https://github.com/OnTheLine/otl-bookdown/blob/master/09.3-bookdown.Rmd.

  • Bookdown, an open-source software package by Yihui Xie at RStudio, allows authors to compose in version of Markdown, an easy-to-read-and-write cross-platform syntax, and create one workflow that produces books in multiple formats, such as Web edition, PDF edition, ePUB ebook edition, and Microsoft Word. Furthermore, Bookdown generates the Web edition as a set of static-site HTML files, which display digital elements (such as interactive maps and videos) and load very quickly into readers’ web browsers. Individual authors can install Bookdown on their Mac, Windows, or Linux computers, and publish their work through a free and simple GitHub Pages account, rather than a separate and complex web server. Interestingly, Bookdown requires installing the open-source R Project and RStudio console and editor, which typically are used for statistical data analysis. Behind the scenes, Bookdown converts the text of the book from R-flavored Markdown files into Web and eBook editions using PanDoc, an open-source universal document converter, without requiring authors to learn complex Pandoc instructions. Although Bookdown may not be the best choice for novice computer users, installation steps are outlined below, along with a recommended workflow for authoring and publishing with GitHub and Zotero.220
  • GitHub is an open-source repository that enables us to publicly share the text and code in this book, with a version-control system that allows collaborators to work on different sections at the same time. Also, the GitHub Pages feature allows us to freely host online all of the book editions (Web, PDF, etc.), with a custom domain name purchased through another service, Reclaim Hosting. As this is an open-access publication, we intentionally make its contents public—and also share instructions like this—while we work on the book, which is consistent with the contract signed with our open-access publisher. Note that GitHub also allows users the option to make repositories private.
  • Zotero is an open-source bibliography manager to collect sources and organize citations, created by the Roy Rosenzweig Center for History and New Media at George Mason University. Better BibTeX is an extension by Emiliano Heynes that makes Zotero citation keys work better with Markdown text workflows. While Bookdown does not require using Zotero, scholars who need to manage multiple source materials and citations will benefit from the Bookdown—Zotero—Better BibTeX workflow described below.

Before leaping into Bookdown or any software tool, see Alternative Publishing Platforms in this book.

Overview of the workflow on a Mac desktop: Compose and build the book in RStudio with Bookdown (top left), manage sources and insert citation keys with Zotero + BetterBibTex (bottom left), and push the built book files to GitHub Pages with settings shown via web browser to share online (right).

Figure 9.1: Overview of the workflow on a Mac desktop: Compose and build the book in RStudio with Bookdown (top left), manage sources and insert citation keys with Zotero + BetterBibTex (bottom left), and push the built book files to GitHub Pages with settings shown via web browser to share online (right).

Install Bookdown with R, RStudio, and a LaTex Engine

Below are steps we followed to setup the publishing platform for this book, using our Macintosh OS 10.14 computers. The same general principles also should apply to Windows computers. No special knowledge is required, but this may not be recommended for novice computer users. Installation steps—and inevitable problems that pop up—will be easier if users are willing to feel adventurous with their computers, or already have some familiarity with text editors, GitHub, or R Studio.

  • Install R Project statistical programming language https://www.r-project.org, which Bookdown requires. See screenshot
  • Install the free RStudio Desktop to make R easier to use with a visual editor. See screenshot
  • Inside RStudio, select the Packages tab, and select Install. See screenshot
  • Inside RStudio, install the “bookdown” package to build your book, and select Install Dependencies. See screenshot
  • Bookdown now should be successfully installed in RStudio. See screenshot
  • For Bookdown to create a PDF edition of your book, you need to install a LaTeX engine to prepare your Markdown plain text, citations, and images into stylized pages. Since the full-sized LaTeX project is very large, Bookdown recommends the smaller TinyTeX package. Inside RStudio, select the Packages tab, select Install, and enter “tinytex” to find and upload the package. See screenshot
  • To finish the installation, in the RStudio console, type tinytex::install_tinytex() and press return. See screenshot
Sidenote: About the PDF edition, and TinyTeX vs XelaTex

The PDF edition of this Bookdown-generated publication is still a work-in-progress, and the settings are not yet finalized. Read more about various options at https://bookdown.org/yihui/rmarkdown/pdf-document.html. To avoid some unicode error messages when generating the PDF edition for this book, we installed XelaTex instead of TinyTeX. If you do the same, be sure to modify this line in your Bookdown index.Rmd settings: latex_engine: xelatex

Sidenote: If TinyTeX installation error
  • Once upon a time, we received this error message when attempting to install tinytex in RStudio on our Mac: /usr/local/bin not writeable. We resolved by reading the software developer’s GitHub issue https://github.com/yihui/tinytex/issues/24 and following these steps:
  • In the Mac Applications > Utilities folder, open the Terminal application.
  • Carefully type sudo chown -Rwhoami:admin /usr/local/bin and press return.
  • Carefully type (without “sudo”) ~/Library/TinyTeX/bin/x86_64-darwin/tlmgr path add and press return.
  • Close the Terminal application.
  • In the RStudio console, type tinytex::install_tinytex(force = TRUE) and press return.

Download and Build a Sample Bookdown Book

  • Create a free GitHub account https://github.com to simplify steps for the next two sections. While Bookdown does not require you to use GitHub, the workflow described below features GitHub to copy a Bookdown template and to host your own Bookdown editions online.
  • In your web browser, log into your GitHub account, go to the Bookdown developer’s bookdown-minimal repo https://github.com/yihui/bookdown-minimal, and fork a copy to your GitHub account. To learn about forking in GitHub, see this chapter http://datavizforall.org/github.html in the Data Visualization for All book.
  • Install GitHub Desktop https://desktop.github.com to transfer files between your online GitHub repo and local computer. While software developers may prefer to access GitHub by typing commands in their terminal, GitHub Desktop provides easier point-and-click access for most users.
  • In your web browser, go to your forked copy of bookdown-minimal, click the green Clone or Download button, and select Open in Desktop. This should automatically open the GitHub Desktop application, and you can navigate where you wish to store a copy of your code repo on a folder in your local computer.
  • In RStudio in the upper-right corner, select Project > Open Project to open the bookdown-minimal folder on your local computer. See screenshot
  • In RStudio, open the index.Rmd file and make some simple edits to the text of this minimal book. For example, remove the hashtag # comment symbol in line 8 to “uncomment” and activate the PDF book option. Save your edits. See screenshot
  • Optional: If you wish, you can modify your bookdown-minimal files outside of RStudio, by using your preferred text editor, such as Atom editor https://atom.io.
  • In RStudio, upper-right corner, select the Build tab, select Build Book, and choose All Formats to build both the gitbook-style static web edition and PDF edition.
  • If RStudio successfully builds both editions of your minimal book, the output will be saved into your bookdown-minimal folder, in a subfolder named _book, because that’s how this sample is configured. The RStudio internal browser should automatically open your web edition (but it’s not a very good browser, so we typically close it and manually open the index.html file with Firefox or Chrome browser.)
  • Also, open the subfolder and inspect the PDF edition of your book. If RStudio found any errors, they will appear in the Build viewer. See screenshot
  • Hint: In future sessions with RStudio, you may need to select Packages tab and click Update to keep bookdown and other software packages up to date. See screenshot
  • Close RStudio.

Host your Book with GitHub Pages

  • Open GitHub Desktop and navigate to the bookdown-minimal folder on your local computer. Write a quick summary to commit (or save) the changes you made above to your master branch, and push this version to your online GitHub repo.
  • In your web browser, go to your online GitHub repo, with a web address similar to https://github.com/USERNAME/bookdown-minimal (insert your own GitHub username).
  • In your GitHub repo, select Settings, and scroll down to the GitHub Pages section, which is a free web hosting service to publish your code and book editions on the public web. Select Master Branch as your source, and Save.
  • Scroll down to this section again, and the web address of your published site should appear. Copy this address.
  • In a new browser tab, paste the web address from above, and at the end add _book/index.html because this sample is configured to store all web and PDF editions in your _book subfolder. Your web address should be similar to: https://USERNAME.github.io/bookdown-minimal/_book/index.html
  • Reminder: You may need to wait up to one minute for edits to your GitHub online repo to appear live at your GitHub Pages web address. Also, after waiting for GitHub Pages to make changes, be sure to “force reload” or “hard refresh” your web browser to update directly from the GitHub Pages server, not the browser’s internal cache. On Mac, press shift + command + R. On Windows, press ctrl + F5.

Customize Bookdown and GitHub Settings and Domain Name

  • To see the file structure, settings, and content for this On The Line book, view its online repository at https://github.com/ontheline/ontheline-bookdown.
  • See details about configuration settings in Yihui Xie’s Bookdown technical guide.221
  • In this publication, the _bookdown.yml file is set to build all of the book outputs (Web edition, PDF edition, etc.) in directory folder named docs in the GitHub repo as shown below. (Proper indenting is important here.)
output_dir: "docs"
book_filename: "Dougherty-etal-OnTheLine"
edit: https://github.com/ontheline/otl-bookdown/edit/master/%s
    fig: "Figure "
    edit: "Edit"
chapter_name: "Chapter "

In this book, the Bookdown setting has been changed from the default _books folder to the docs folder to take advantage of the GitHub Pages setting that allows hosting online content from the master/docs folder, rather than the entire master branch. Choosing these Bookdown and GitHub Pages settings allows authors to publicly share their book source code and raw text on the master branch, and host online the Web and ebook editions from the master/docs subfolder of that branch. (Years earlier, GitHub previously required users to separate these tasks on two different branches, typically called master and gh-pages, but now users can do both through one branch, which is simpler.) Also, following this master/docs method simplifies the address of your GitHub Pages published web edition to this format: https://USERNAME.github.com/REPONAME. (See further below about customizing the web address.)

In this publication, the index.Rmd file is set to produce the Web edition in the GitBook style, with additional settings explained further below. (Proper indenting is important here.)

    dev: svglite
    css: css/style.css
      in_header: [custom-scripts.html, google-analytics-otl.html]
    split_by: section
    split_bib: true
    number_sections: true

In the GitHub repo for this book, here is the root file structure:

  • Preface of the book with non-numbered sections: index.Rmd
  • Chapters with first-level headers, such as: 01-introduction.Rmd
  • Sections with second-level headers, which belong to chapters, such as 01.1-section.Rmd and 01.2-section.Rmd
  • The images folder, where JPG, PNG, and PDFs to display in chapters are located.
  • The docs folder, which contains the published book products, such as Web edition (index.html, introduction.html, etc.), the PDF edition (Dougherty-etal-OnTheLine.pdf), etc.
  • Additional helper files described further below.

When you change the names of chapters/sections, Bookdown builds new HMTL pages based on those new names, but old HMTL pages based on old chapter/section names may still exist in the same subfolder. To avoid confusion, you may wish to carefully delete old HTML pages in docs whenever you significantly alter names and build a new version of the book.

In addition, this GitHub Pages repo is published with a custom domain name https://OnTheLine.trincoll.edu. Learn more about custom domain names at https://help.github.com/articles/using-a-custom-domain-with-github-pages/. Typically, a custom domain name is purchased from a web hosting service, such as ReclaimHosting, and you may need customer support to access the Manage DNS (domain name system) to point your new URL to your GitHub Pages online site. For this book’s custom domain name, https://ontheline.trincoll.edu, the IT staff at Trinity College created a subdomain on their .edu domain, which shows that my scholarship is affiliated with an academic institution. Adding a GitHub Pages custom domain name creates an additional CNAME file in the docs subfolder. Avoid accidentally erasing the CNAME file if you need to clear out any old HTML files in your subfolder.

The web edition of this book includes two custom files that are referenced in the index.Rmd file header, and are generated into the web edition when building the book:

    dev: svglite
    css: css/style.css
      in_header: [custom-scripts.html, google-analytics-otl.html]
  • the custom-scripts.html file includes specific instructions to set the height or width of iframes for specific interactive maps and charts in the web edition, which is explained further in the Style Guide section of this book. (Thanks to Michael Dorman for this tip.)
  • the google-analytics-otl.html file records web traffic with Google Analytics, if you create a free account and insert your code.
  • The web edition of this book also includes a custom 404.html file that redirects any bad web addresses under the domain back to index.html page. A copy of this file must be manually transferred into the docs subfolder, because it is not automatically built by Bookdown.

In this book, the index.Rmd file also contains instructions to create footnotes and a bibliography from a Zotero-Better BibLaTeX exported file named ontheline.bib, in a specific Chicago-style format. Other .csl files are downloadable from the Zotero Styles Repository https://www.zotero.org/styles.

bibliography: ontheline.bib
citation-style: chicago-fullnote-bibliography.csl

Also in this book, the index.Rmd file is set to produce a PDF edition of the book using the xelatex engine (or see tinytex or latex alternatives described above). Furthermore, additional PanDoc settings are required to create the desired Chicago-style footnotes and bibliography in the PDF edition. See more details at https://pandoc.org/.

    latex_engine: xelatex
    citation_package: none
    pandoc_args: [ "--csl", "chicago-fullnote-bibliography.csl" ]

Zotero and Better BibTex for Footnotes and Bibliography

This workflow uses the open-source Zotero bibliography manager (http://zotero.org), with the Better BibTeX extension (https://retorque.re/zotero-better-bibtex/) to work more easily with Markdown and Bookdown. Rather than typing full references directly into the text, authors can insert a short citation key to each source, and the tools will automatically generate the desired references (such as Chicago-style footnotes, which historians love) and an alphabetized bibliography for the book. Furthermore, since authoring this On The Line book has been a multi-year process with multimedia sources whose web addresses have changed over time, this workflow ensures that the most up-to-date information in your Zotero database also appears in all editions of your book.

With these tools installed, here’s an outline of the author’s workflow: - An entry for each source (book, journal article, document, etc.) must be created in the author’s Zotero library (or a specific collection), either automatically from the web or manually. - Better BibTex generates a unique citation key for each source (example: ^[@tyackOneBestSystem1974]), which authors insert into the Markdown text to create a full reference (such as a footnote) and a bibliography. - The author identifies their preferred citation style at https://www.zotero.org/styles, uploads the file (example: chicago-fullnote-bibliography.csl) to the local book repo, and lists it in the index.Rmd settings for both the Web edition and the PDF edition. - Before each build in Bookdown, the author sets their Zotero preferences, and exports the Zotero library or specific collection to the root level of the local book repo (example: ontheline.bib).

Installation and Settings
  • Download and install Zotero for Mac, Windows, Linux, and add connectors to your browsers.
  • Install the Better BibTeX extension and follow all of the site’s instructions for initial setup.
  • At the top of each entry in Zotero, the extension will generate a unique citation code (example: tyackOneBestSystem1974).
  • You can copy and paste the citation code into your Markdown text, and add a caret, square brackets, and the at symbol ^[@tyackOneBestSystem1974], as described in the Style Guide in this book.)
  • Also, you can set Zotero preferences > Export > Better BibTeX Quick Copy to use Zotero’s drag-and-drop quick copy feature.
  • To export a bibliography (.bib) from Zotero library or collection into the local Bookdown repo:
    • Select Library > Right-click to export the collection
    • Select format > Better BibLaTeX (IMPORTANT I use this setting, rather than “Better BibTex”, because “Better BibLaTex” includes full dates in newspaper citations, and urls). Also, I leave all of the checkboxes blank during the export, and do NOT select “keep updated”. This means that if my Better BibTex citation codes suddenly change in Zotero because the author, title, or year changed, then I’m responsible for running find-and-replace to make these edits in the text of the book.
    • Save the output as FILENAME.bib, save into your book repo, and be sure to match the settings in index.Rmd.

These Zotero types appear correctly with this Bookdown workflow: - Book - Book chapter - Journal article - Newspaper - Thesis - Report - Blog post - Web page - Document – Use this all-purpose entry in place of other types: Law case, Presentation, Interview, Video recording, Television broadcast, etc. Insert important details (such as the archival location information) in the Publisher field.

To help other researchers find items cited in this book, URLs should be included in Zotero entries whenever feasible, even if not required by convention. For example, some print-only books and documents are hard to locate, so include an OCLC WorldCat permalink to make them easier to find (example: http://www.worldcat.org/oclc/20683509). Also, if a print source has been digitized by HathiTrust, Google Books, or the Internet Archive, add one of these URLs to the Zotero entry.

Reminder: Chicago full-note works exactly as it was designed, meaning that the second instance of a citation currently appears as an abbreviated note (author, with title when appropriate). TODO: Find or create a modified .CSL file to display the full footnote entry in every instance in the web version, so that users do not need to scroll to find the complete source info.

On The Line is an open-access, born-digital, book-in-progress by Jack Dougherty and contributors at Trinity College, Hartford CT, USA. This work is copyrighted by the authors and freely distributed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. Learn about our open-access policy and code repository and how to read and cite our work. This book-in-progress was last updated on: 2019-08-13

  1. For a detailed guide, see Yihui Xie, Bookdown: Authoring Books and Technical Documents with R Markdown, 2018, https://bookdown.org/yihui/bookdown/

  2. Xie