Reference Frames: Higher values yield better quality videos, but also take more time to encode. Set it too high and you could run into playback issues.
Mixed References: Video frames are divided into blocks called 'macroblocks.' Checking this box (and thus turning Mixed References on) allows different references to be used for different parts of each macroblock, potentially resulting in a higher quality rip.
B-Frames: Stands for bi-directional frame and is a compression method. Handbrake's developers recommend using a higher value (9-16) for animated content.
Adaptive B-Frames: Allows x264 to determine the number of B-frames to use.
Direct Prediction: Tells x264 what method to use when guessing motion for certain parts of a B-frame. If set to spatial, x264 will compare parts of the current frame, whereas temporal tells x264 to compare against the following P-frame. Handbrake's developers recommend choosing spatial over temporal.
Weighted B-Frames: Rather than base a B-frame's motion compensation on frames both and after with equal influence, weighted B-frames takes into consideration the distance from the frame being encoded.
Pyramidal B-Frames: As a High Profile feature, Handbrake's developers warn that enabling this setting will prevent your video from playing in Quicktime. What it does is treat each previous B-frame as a reference, resulting in higher quality, lower bitrate rips, but "at the expense of complexity."
Motion Estimation Method: Selects the way motion is detected by x264, ranked in order from the fastest encode time (at the expense of quality) to slowest, with Diamond being the fastest.
Subpixel Motion Estimation: Determines how x264 makes decisions about motion estimation, with higher levels increasing quality, and lower levels resulting in faster encode times.
Analysis: Determines how to divide up frames to capture details. In most cases, you'll want to leave this on "all."
8x8 DCT Blocks: Another High Profile feature, checking this box tells x264 to break frames down into 8x8 blocks for slightly better quality rips.
Trellis: When used with multi-pass encoding, this is supposed to improve quality, but can also significantly slow down encoding.
CABAC Entropy: Short for Contact Adaptive Binary Arithmetic Coding and is capable of reducing the bitrate. Not compatible with some popular devices.
No Fast P-Skip: Improves encoding speed by blocking on-solid colors, but also reduces quality.
No DCT Decimate: Tells x264 to "zero out" blocks when they won't be noticed by the viewer. Can be useful in reducing file sizes, but can also introduce unintended artifacts.
Deblocking: Smoothes out blocky compression artifacts. The higher the value, the more aggressive it works.