The public flag only affects search.  When the public flag is set on a document, if a search matches on the document, the document will show up in the search results regardless of if the user has access to the document or the workspace containing the document.  The search results will only show that there was a hit but will not show the contents of the document so as to protect the contents from trolling.

The design of the public flag is for large organizations who want to create a separate organization for past performance and drop the final copies of proposals into that organization.  In this case you don't want to give everyone access to all of your past documents but you want them to be able to determine that there is a document that may be useful for a current effort.  In this scenario a user might do a search and identify that a document exists that may be useful and they can then right click on the search results to ask the owner for access to the document.