I discovered this technique quite by accident. What I was trying to do was to create a rock formation that hung over at more than 90 degrees. Creating a terrain in Bryce, stretching it vertically and just rotating it on the X or Z access just didn’t work. There were scaling artifacts and hiding the back edge that was now poking out was not acceptable.
Then I started to fiddle with the Scaling options in the Edit Palette and discovered that if you use your Rotate and Scale functions with the Options set to "World Space" they scale and rotate according to the world coordinates and not the object itself.
1. Create a terrain – preferably one that has a rocky outcrop in the middle of the terrain object – like an island in the middle of the sea.
2. Go to your Edit Palette and click on the small white arrow just below the Rotate tool. Ensure that "World Space" is selected from the Rotate options. (If you change this option in the Rotate tool it will also change the option in the Scale and Move tool as well)
3. Now change your view to a look at your terrain from the side. It will help to give you a better idea how much you should rotate. Now rotate the terrain about 30 degrees. Whether you rotate clockwise or anti-clockwise is up to you. Keep in mind that the overhang will occur on the side that is opposite to the direction in which you are rotating.
4. Now go to the Scale tool and click on the White arrow to get the options popup menu. Check to see if the option is still set to "World Space".
5. Now scale the terrain on the Y axis and because the scale tool is set to "World Space" the world’s Y axis and not the terrains Y axis will be used.
6. Now rotate the terrain back on the X axis so that the base is parallel with the ground plane. You will now see that the top of your rocky outcrops are leaning over more than 90 degrees and you keep the detail on the rock surface.
7. Finally you can use the "Land Object" tool – the small arrow at the bottom of the object controls to the right of your terrain. Render and see your results – a rather nice-looking overhanging rock!
I found that the more you rotate the object originally, the more extreme to overhang will be. I also experimented and discovered that terrains with peaks concentrated in the middle of the terrain often work better if you are looking for an isolated rock. If you have created a terrain in the Terrain Editor that has some details or "high spots" on the edges of your map, use the Gaussian Edges or the Round Edges function in the Terrain Editor to lower and remove those edge details.
Here is the result of my experimentation. Many thanks to my friend Roy Hindes for reminding me about this "World Space" option!