Inspect data in BoltDB

Vault clusters configured to use Integrated Storage (Raft), the cluster persists all encrypted data to a bbolt key/value store.

You can inspect data in clusters which use Integrated Storage by operating Vault in recovery mode and using the sys/raw API as described in Inspect Data in Integrated Storage. That process is useful for troubleshooting issues in operable Vault clusters and provides the greatest detail possible for issue resolution and troubleshooting.

You can also use information in this tutorial to access low level details from the database by directly examining the database files.

This is a tool of last resort, but it's helpful when you suspect database corruption, when a Vault cluster is inoperable, or for querying the free list.

Prerequisites

To follow the examples in this tutorial, you need the following.

• A Go development environment to install the tools

Learn monitoring repository

You can get some example Vault database files to use when following along in this tutorial by cloning the learn-vault-monitoring repository or downloading a zip file from GitHub.

1. Clone the repository:

   $ git clone https://github.com/hashicorp-education/learn-vault-monitoring.git
   

   Copy

   Or, download the repository contents:

   

   Tip

   If you downloaded the zip archive, be sure to unzip the file before proceeding.

   This repository has supporting content for Vault learn tutorials. The content specific to this tutorial is in a subdirectory.

2. Change the working directory to learn-vault-monitoring/example-raft-database.

   $ cd learn-vault-monitoring/example-raft-database
   

   Copy

3. Examine the directory structure of a data directory from a Vault server configured to use Integrated Storage.

   $ tree
   .
   ├── raft
   │   ├── raft.db
   │   └── snapshots
   └── vault.db
   

   Copy

   This is the most minimal example, with a raft directory containing the cluster coordinating data in raft.db, and a vault.db that holds the Vault operational data and secrets.

   Were this from a busier cluster, the snapshots directory would also contain some snapshot data.

With your example Vault data in place, you are now ready to inspect the files with the bbolt CLI and via a web browser with boltd.

Notes about storage architecture and terms

bbolt uses a database file represented by pages. The first 2 pages store important database metadata, while the third page holds the "free list". The free list keeps track of pages marked as deleted, which are free for writing to again.

All remaining pages contain the buckets. All keys and values get stored in buckets, represented by B-tree data structures.

Using the bbolt CLI

The bbolt CLI tool is part of the bbolt distribution.

1. Install bbolt into your $GOBIN path where GOBIN is the directory where go install and go get will place binaries after building main packages.

   $ go install go.etcd.io/bbolt/cmd/bbolt@latest
   

   Copy

2. Check the bbolt installation to make sure it's on your $PATH.

   $ bbolt help
   Bbolt is a tool for inspecting bbolt databases.
   
   Usage:
   
     bbolt command [arguments]
   
   The commands are:
   
       bench       run synthetic benchmark against bbolt
       buckets     print a list of buckets
       check       verifies integrity of bbolt database
       compact     copies a bbolt database, compacting it in the process
       dump        print a hexadecimal dump of a single page
       get         print the value of a key in a bucket
       info        print basic info
       keys        print a list of keys in a bucket
       help        print this screen
       page        print one or more pages in human readable format
       pages       print list of pages with their types
       page-item   print the key and value of a page item.
       stats       iterate over all pages and generate usage stats
   
   Use "bbolt [command] -h" for more information about a command.
   

   Copy

   Tip

   If you meet with an error instead, ensure that the go install succeeded and that bbolt is in your $PATH with which bbolt.

Check database file integrity

One useful feature of bbolt is the check command. It verifies that all pages are accessible or marked as freed, and also checks that no pages are double referenced.

If you suspect issues with the database file, the first thing you can do is check its integrity.

$ bbolt check vault.db
OK

The expected successful result is OK.

If there are issues with the database, the tool will emit errors.

Statistics

You can also gather essential statistics about the database.

$ bbolt stats vault.db
Aggregate statistics for 2 buckets

Page count statistics
  Number of logical branch pages: 11
  Number of physical branch overflow pages: 0
  Number of logical leaf pages: 297
  Number of physical leaf overflow pages: 135
Tree statistics
  Number of keys/value pairs: 1314
  Number of levels in B+tree: 3
Page size utilization
  Bytes allocated for physical branch pages: 45056
  Bytes actually used for branch data: 22775 (50%)
  Bytes allocated for physical leaf pages: 1769472
  Bytes actually used for leaf data: 1238220 (69%)
Bucket statistics
  Total number of buckets: 2
  Total number on inlined buckets: 1 (50%)
  Bytes used for inlined buckets: 317 (0%)

The stats command performs an extensive search of the database to track every page reference. It starts at the current meta page and recursively iterates through every accessible bucket. The command will emit errors if the database is corrupt.

From the example output, you learn that this database consists of 1314 key/value pairs and that there have been 1238220 bytes actually used for the leaf data.

More information is available from the help output with bbolt stats -h.

List buckets

The Vault database starts with 2 buckets: config and data.

You can list them like this:

$ bbolt buckets vault.db
config
data

List and get keys

The goal of this section is for you to learn how to list keys in more detail.

1. List all keys present in the config bucket.

   $ bbolt keys vault.db config
   latest_config
   latest_indexes
   local_node_config
   

   Copy

   The keys in the config bucket contain information about the latest cluster configuration, including a list of cluster members, latest index state, and whether the local node is a voter.

2. Get count of the keys in the data bucket.

   $ bbolt keys vault.db data | wc -l
       1311
   

   Copy

   The data bucket holds the majority of all data in Vault, and should also represent the majority of the keys found in the database.

   Since there are several keys present in this example, you can explore a smaller subset of the keys contained in the data bucket.

3. List just the keys which make up the Vault core information.

   $ bbolt keys vault.db data | grep core
   core/audit
   core/auth
   core/autoloaded-license
   core/cluster/feature-flags
   core/cluster/local/info
   core/hsm/barrier-unseal-keys
   core/index-header-hmac-key
   core/keyring
   core/leader/dceb963e-72a2-f1d4-ea8f-c903dbb57dd1
   core/local-audit
   core/local-auth
   core/local-mounts
   core/lock
   core/master
   core/mounts
   core/raft/tls
   core/seal-config
   core/shamir-kek
   core/wrapping/jwtkey
   

   Copy

   These keys represent the Vault core, and Vault uses them internally to contain critical configuration information.

4. A more practical example than demonstrating exploration is to find the count of valid tokens in the current Vault data. You can do so by counting the entries under the path sys/token/id/.

   $ bbolt keys vault.db data | grep 'sys/token/id/' | less | wc -l
       228
   

   Copy

5. Another practical example is confirming the existence of a particular policy by name. If for some reason you need to verify that the cluster has a particular named policy, you can list all names.

   $ bbolt keys vault.db data | grep 'sys/policy/' | cut -d '/' -f3
   admins
   control-group
   default
   lab-user
   ldap-dev
   ldap-user
   prometheus
   response-wrapping
   vaultron-example-namespace-ns
   vaultron-example-root-ns
   vaultron-example-token-admin
   vaultron-example-token-identity
   vaultron-sudo
   

   Copy

   Note

   Along with listing keys, you can get key values for a limited subset of keys which allow unauthenticated access. Vault encrypts all user supplied inputs at rest, and you will not be able to decrypt them using the techniques and tools in this tutorial.

   One such exception is the list of cluster members found in the latest_config key in the config bucket.

6. Get the latest_config key values.

   $ bbolt get vault.db config latest_config
   }%vaultron-vault-010.10.42.200:8201%vaultron-vault-110.10.42.201:8201%vaultron-vault-210.10.42.202:8201%vaultron-vault-310.10.42.203:8201%vaultron-vault-410.10.42.204:8201
   

   Copy

   These are the 5 servers which made up the Vault cluster these data are from.

If you have concerns about your terminal session, you can also try testing the output with file. Using the earlier examples from listing keys making up the Vault core, you can check if it might be safe to view the server's seal configuration in core/seal-config.

$ bbolt get vault.db data core/seal-config | file -
/dev/stdin: JSON data

This is JSON data that provides some helpful information about the server's seal configuration.

$ bbolt get vault.db data core/seal-config
{"type":"shamir","secret_shares":1,"secret_threshold":1,"pgp_keys":null,"nonce":"","backup":false,"stored_shares":1}

While generally not as useful as the vault.db, you can also inspect data in the raft.db. For example, to find the cluster member which was the last candidate voted a leader, you can check the LastVoteTerm key in the conf bucket.

$ bbolt get raft/raft.db conf LastVoteCand
10.10.42.201:8201

Pages and the free list

This section is most useful for users of Vault 1.8.0 and higher who wish to learn more about the free list status.

1. To begin, list the first 2 pages.

   $ bbolt pages vault.db | head -n4
   ID       TYPE       ITEMS  OVRFLW
   ======== ========== ====== ======
   0        meta       0
   1        meta       0
   

   Copy

   The list has 2 meta pages. You can use the most current one with the highest transaction ID value to find the freelist location.

2. Check the transaction ID for page 0.

   $ bbolt page vault.db 0 | grep -B2 'Txn ID'
   Freelist:   <pgid=306>
   HWM:        <pgid=477>
   Txn ID:     1712
   

   Copy

3. Check the transaction ID for page 1.

   $ bbolt page vault.db 1 | grep -B2 'Txn ID'
   Freelist:   <pgid=174>
   HWM:        <pgid=477>
   Txn ID:     1711
   

   Copy

   Page 0 has the highest Txn ID value, and is the meta page you should examine to find the root of the tree. Note that its Freelist value points to the correct freelist page ID.

4. List page 0 to find the root page ID.

   $ bbolt page vault.db 0 | grep Root
   Root:       <pgid=235>
   

   Copy

5. The root page ID is 235, and from here you can walk the entire tree of pages. Begin by listing page 235.

   $  bbolt page vault.db 235
   Page ID:    235
   Page Type:  leaf
   Total Size: 4096 bytes
   Item Count: 2
   
   "config": <pgid=0,seq=0>
   "data": <pgid=212,seq=0>
   

   Copy

   The output shows the pages representing the config and data buckets.

6. You can retrieve the data page at 212.

   $ bbolt page vault.db 212
   Page ID:    212
   Page Type:  branch
   Total Size: 4096 bytes
   Item Count: 10
   
   "audit/4487ee65-4449-6a32-d0af-518c380a0ede/salt": <pgid=139>
   "index-dr/pages/99": <pgid=58>
   "sys/expire/id/auth/userpass/login/lab-user-1/   h04a2ea39ac9f9b5fca932aab0f7ed25140db7e06e620926ffb43530a6911b61b": <pgid=162>
   "sys/expire/id/auth/userpass/login/lab-user-3/   h999915ecf7b1956d2287c913979ebe3ae6e2d35cca151f7da5e840ecf56b212d": <pgid=302>
   "sys/expire/id/auth/userpass/login/lab-user-5/   h74f9c9511f932e06e8f3e574947dd0e8f7279f2cc50cc3d3957320d539e2cfe7": <pgid=221>
   "sys/expire/id/auth/userpass/login/lab-user-7/   h44ba7953b65aceec11658b9371546743162fa86f999d7c86ecb3ce39abfa42e7": <pgid=217>
   "sys/expire/id/auth/userpass/login/lab-user-7/   hc6386db098e528405526f7b288703038a0358c4e90372e8f5b77e6f62e8b4222": <pgid=301>
   "sys/token/accessor/d1db3bdfd1cae11be408a41157996f32646f62af": <pgid=281>
   "sys/token/id/h60a215b77e0bc911cae8e988de36a8c1df736c57cf0e9023c0bb2430ea8538f3": <pgid=196>
   "sys/token/id/he32264f2dc56be09df07c4ed7b2b45987b440e459c9c622183527ddbf4ca41dc": <pgid=194>
   

   Copy

7. Page 162 represents a username and password auth method lease id value. You can list this page ID to view its contents.

   $ bbolt page vault.db 162
   Page ID:    162
   Page Type:  branch
   Total Size: 4096 bytes
   Item Count: 17
   
   "sys/expire/id/auth/userpass/login/lab-user-1/h04a2ea39ac9f9b5fca932aab0f7ed25140db7e06e620926ffb43530a6911b61b": <pgid=64>
   "sys/expire/id/auth/userpass/login/lab-user-1/h4a13e2e920de1a69720984c2c643b172d7a451bcbf4d1bd90b1d5ba3bc56ed2d": <pgid=98>
   "sys/expire/id/auth/userpass/login/lab-user-1/h9f7b901d7bddc6407d82ef442c79bb6407804c5717c999ea74cc81a434f4efde": <pgid=41>
   "sys/expire/id/auth/userpass/login/lab-user-1/he93aa7f0493831dbdb5fada98f790f6b66eb1cbde985dbc49962629b7cd95fe0": <pgid=125>
   "sys/expire/id/auth/userpass/login/lab-user-3/h021e7aadf3008a2bb47ea917a6de8402439bd4637b78b11d56b372f5f1872627": <pgid=169>
   "sys/expire/id/auth/userpass/login/lab-user-3/h0cf7e2818d1f43250793cc3adf37d2cd65ea6421fe2ca1411b8ed59885f23b8b": <pgid=53>
   "sys/expire/id/auth/userpass/login/lab-user-3/h16a29ff005344742a5ca44a27d04bcc26467012229b9aba5977266e4a7e5888c": <pgid=118>
   "sys/expire/id/auth/userpass/login/lab-user-3/h20fef3012a68b1f3d9a0e3f42d08d5f22dfc9d5584582777538e8be1aeda9d78": <pgid=198>
   "sys/expire/id/auth/userpass/login/lab-user-3/h32be213efb25de70f7401a6955a3b716f0e4dcddc654629edfacaa5754742e41": <pgid=134>
   "sys/expire/id/auth/userpass/login/lab-user-3/h4109fc3b1d299624d81eabc9d7e5be4e0bf03d7bfe28d4b4a1de439b99f0f03e": <pgid=155>
   "sys/expire/id/auth/userpass/login/lab-user-3/h44e2ad5e382b80ceb2ab45d10710d9feb1468d0baf853c7f94df57e2b0db7aa8": <pgid=4>
   "sys/expire/id/auth/userpass/login/lab-user-3/h494323e1dcad8ae427e163d0fffcc6bbf317c8a6d9c3489d98d3cf22fe98cfdb": <pgid=181>
   "sys/expire/id/auth/userpass/login/lab-user-3/h5d7b4c21ae89c98952a45bb6bbae4698f2a1c0d7715022e2f9456092ab423b7f": <pgid=105>
   "sys/expire/id/auth/userpass/login/lab-user-3/h69290ffe0e01a35568c381ac6b4a6cea66c2947d30c4266d8410a45f65794216": <pgid=66>
   "sys/expire/id/auth/userpass/login/lab-user-3/h7396c835ae280c4115c01835e5f83b072eafef5480e758fe254bc7135fd610a2": <pgid=138>
   "sys/expire/id/auth/userpass/login/lab-user-3/h78b7a2a0688a3f6afe17e6d34cb9b96cbf926fc26a891300c220390fbc5adba8": <pgid=122>
   "sys/expire/id/auth/userpass/login/lab-user-3/h885ab1f14922f3310924bac61019c8a894dc3d55bbb1fa98f1cc71ddb1e2d318": <pgid=123>
   

   Copy

8. Now you can view the leaf page for an individual lease id, for example the one at page ID 64.

   $ bbolt page vault.db 64
   Page ID:    64
   Page Type:  leaf
   Total Size: 4096 bytes
   Item Count: 3
   
   "sys/expire/id/auth/userpass/login/lab-user-1/h04a2ea39ac9f9b5fca932aab0f7ed25140db7e06e620926ffb43530a6911b61b": 000000010232b3d254a9581ad479364b7ba235aa11a91bc225adc8c447e0ec267ba2c489492ff4d5df7891aa3fc91de61cd30bab462a91674f13117f613e9f6ae7bb1e903eaaed485db5bde4e98debd601a0f5303fcc5f0244df3d4795f1b5a5db7fb8ff9101e8dabd43b092902043af6dc59755acafe0730850410d37c8717f65cd09e015c15ddc665515b37ca8be8f51d67abc7ab093183a8d11a4c2edf56bce6a53691e0b066c7b2c37ff3d7062169850f79fa8d89602ad4b6bb5fd3c7d741939fa8bc3dd84373a1e86f79ebabcc1ff91c38728d57806483d2d4e03d6aef414ba0c8826ddc9b4d953220c7dda59cde87b71a4db778e365dd41f27e6d0f7089074aa82eb0ced12d5cf3cd9c241094e0ba034fdaab13f264adf4a73341a3944c08b16ba7581cc5c546cfea21909adde5792530aec8c45d6b3a491541f48ad194164a8ffd773908247c7f649f9d07e962a2addabc2742aaea6864417ae8e3e86a914652f2f72cc1b8abab2be7348208c46ca485887a7bb12579ea0e6e13f6d61ada5f339f1a5f2fe8999a46efb05ce9582d942094e2055828642c69e67648de867d9e1e14bdfc5f526e10c569de74e695b1c58e649ce50d02eab8268456010a3126fc079012af2dfede424913fcd756ea1f7498a7eb584f03bfc59b408a3205bbed61c931131cd2ae67a2dd985439a2924efb73f9b49ba7acdeea23e67226fbc4225f158e2975b7cd363661f7efb1ee3ae2ac7ff0930e2af8fe59478920f3b86fd840dd1280c901d2ad1ea174c850b893612d58fcb5d45a32e5e08a1167925a2a506d57aef0c0e0cf5d11205d346737904dfa3f5c89c624728241d7b98176a26dc8deb5e6e7e9ddbe6df7847922d7976920ad699f79a94c2465627f8b41b1680170ad2b5c07d872739b73e78f3b3b44a8b9a00438316c611d66d99ec6db0bc07522324d3b1c9be79f76cfd179e6b2e65a3afc771c3e9a84dac58471fc1e54390b00a3c4f0166e67fce6ff121527abe45b363e63404c6f0290687bdc7fa1ff29ae61f35f4bad69ccadc71da8e2af4632ac565e931cfa0ad4919fa802a2c1b5e557796b9e8ca4b93b25b2c991a87079c8d18fc03debb416d8be6ae3f4d10b49f65fef46415428c14111d4f20a767b1218da34f7f3343bde1268461e547fd74d8841cd0fc22acd1116dda4e4be9d8bbd100b125318e22ec04568ba2603c9aebc249b9fdc7837557d87bf77ff8a496c159a25bd531c3f5fcb5a11267f52c1fdfd51bc330fb3ed84903d7755850be54fa7cdf9726a9da77e330f1dd934d84c3e6b4afe13d4ba574fe95695ece561c8e9b936b56ff905fc045b07c9bc2494bf48ce60588bef4ec5b213384ad97af5e9a82dc3ffdd8721931a30aa707262d8ef796f300eed4eb3f89d2ae7b0586a4d4a889ccb55d29ce348a517fe40622b1addb8e2f24d92b0271ea37bd7674de741749af6ded672abcbc5d238aad4f39c8496f996db894cb9029ab8fbbe4b7c29924b67606daaef0f2f686cc5362544306f1fba53d380f1ed5777e8f38d47e9fbf011f45
   ...snip...
   

   Copy

   The result is a key sys/expire/id/auth/userpass/login/lab-user-1/h04a2ea39ac9f9b5fca932aab0f7ed25140db7e06e620926ffb43530a6911b61b with a long hexadecimal string representing the encrypted value.

   You can explore the entire tree this way.

9. With the freelist itself, recall that meta page 0 was the page ID containing the most recent Txn ID. Check the Freelist value for meta page 0.

   $ bbolt page vault.db 1 | grep -B2 'Txn ID'
   Freelist:   <pgid=174>
   

   Copy

10. The freelist page ID is 174. Go ahead and list it next.

    $ bbolt page vault.db 174 | head -n 15
    Page ID:    174
    Page Type:  freelist
    Total Size: 4096 bytes
    Item Count: 31
    Overflow: 0
    
    57
    58
    151
    189
    194
    212
    235
    306
    313
    

    Copy

    Note the presence of 31 pages in the freelist.

    The other way to verify a page is free, is to check for its page ID in the pages output, and ensure that free appears next to it.

11. Confirm that page 313 is free.

    $ bbolt pages vault.db | grep -w 313
    313      free
    

    Copy

Using the boltd web UI

Another tool from the original BoltDB project, boltd exists to explore the database visually.

You can use boltd by attaching it to a Vault database and connecting a web browser to the web UI it exposes.

1. Install boltd.

   $ go get github.com/boltdb/boltd/...
   

   Copy

2. Start boltd and attach it to vault.db.

   $ boltd vault.db
   Listening on http://localhost:9000
   

   Copy

3. Access http://localhost:9000 in a browser.

   

   The user interface is minimal, and defaults to displaying a view of pages.

   You can use it to browse facets of your Vault data like the bbolt CLI tool, by navigating links or jumping directly to page numbers in the Go to page input.

   You can also explore a page usage histogram by selecting Show Page Usage.

   

Summary

You have learned about how to inspect data in the bbolt database file for a Vault server using the Integrated Storage backend. You learned about 2 tools which you can use for this purpose along with caveats around their usage.

You can learn more about inspecting Vault data in Inspect Data in Integrated Storage and Inspecting Data in Consul Storage.

Clean up

When you finish the hands on lab, you can clean up by changing to the parent directory and removing the learn-vault-monitoring directory.

$ cd ../..

rm -rf learn-vault-monitoring

Help and reference

• bbolt
• boltd