Sometimes the words they represent are themselves slang terms. For example Ben Hur for stir (prison). Longer phrases are often shortened to the first word, adding to the confusion: blowing raspberries comes from raspberry tart for fart. Both these principles are found in the phrase put up your dukes as a challenge to a fist fight. Dukes is short for Duke of York for fork which is a slang term for hand and by extension fist.
Some words have several slang terms to choose from. Depending on the situation, you can refer to your wife variously as light of my life, carving knife, or trouble and strife. Again usually shortened as in "I'll see if my trouble is home". Money can be bees and honey, bread and honey, or Bugs Bunny. The shortened form bread was adopted by the hippies in the 1960s, so was one of only three in the book that I recognized (but had no idea of the origins).
Names are commonly used as slang terms. Most are real people, well-known (at the time anyway), and may or may not have any connection to the word they represent. Richard Burton for curtain makes sense. Captain Scott (who froze to death in his 1912 South Pole expedition) ironically represents the word hot. Tom Sawyer (a well-known fictional character) replaces lawyer, with no obvious connection other than the rhyme. Other names are simply made up to rhyme such as Mrs. Duckett for bucket and Harry Huggins for muggins (slang for idiot).
These slang terms changed considerably over the years, with modern versions replacing outdated ones. There is also great regional variation, as you would expect when anyone at any time could make up their own.
To test how readily a new rhyming slang phrase could be understood by its context, I'll make one up just for you: I hope you enjoyed this hog roast!
Source: Cockney Rabbit - A Dick'n'Arry of Rhyming Slang by Ray Puxley, 1992.