The best approach for your situation depends on your answer to this question: do you need to access in-memory state?
If the answer is NO*, you can create a new instance of your ActiveX control to do the work. The popular ieSpell extension provides an example of this type of interaction. (I hope they don't mind me using them as an example.) First, their context menu option is registered against a resource that has been compiled into their DLL.
The compiled resource contains the following script:
Notice how they also pass a VARIANT argument to the ShowOptions function (external.menuArguments) that contains the IDispatch of the active window. If you also choose to do this, remember to use VariantClear to release this value.
* If the answer to my earlier question is YES, the problem becomes a little more difficult. One way to address this is to store a pointer to the BHO in a place where it can be retrieved later (presumably by script). IWebBrowser2 provides such a mechanism, PutProperty and GetProperty. Unfortunately, script cannot access this pointer natively because it doesn't have access to the IWebBrowser2 interface of the browser; however, an ActiveX control can. Controls are sited with IObjectWithSite, meaning they are passed a pointer to the IWebBrowser2 interface of the host, and have access to stored properties. Hence, a control can call the BHO directly or act as intermediary between the host and the script. In this latter case, the BHO must be prepared to handle calls from script through an IDispatch interface.
Note: there are at least two more ways to accomplish "advanced" communication with a BHO. They are a little more complicated, so I'm saving them for another time. Let me know if either of the approaches I have presented so far sound reasonable to you.