There isn't a specific capability for home group. It exposes other libraries (i.e pictures, music, or videos) in the home group based on those capabilities. See
http://msdn.microsoft.com/en-us/library/windows/apps/windows.storage.knownfolders.homegroup.aspx
The documentsLibrary capability is restricted and not recommended. Can you explain why and how your app needs o use the documents library?
In almost all cases you will be better off letting the user choose a location with a file or folder picker rather than directly using the documents library, and if you plan to submit the app to the store it probably won't be certified with the documents
library capability.
If you really donned it you will need to hand edit the manifest file.
See
App capability declarations (Windows Store apps)
If you need more help please explain your scenario and we can help you find the best way to achieve it.
--Rob