JTAPI has no ability to effect the phone's audio systems: e.g. volume, mute, speaker, headset, etc.
Would putting the call on hold (or doing something like Redirecting it temporarily to a CTI port that plays silence), suffice?  That can of course be done via JTAPI.
 

sendData is an interesting capability, and there is a full XML API for utilizing the phone's display and certain other features by apps in this way: http://developer.cisco.com/web/ipps/home

Mohan's suggestion is to use the XML API's ability to simulate physical button presses on the phone - via the Key:??? URI - to simulate pressing the phone's Mute button.  Unfortunately there is no corresponding API for reading the current state of the Mute button, so this button push is 'blind' and may do the opposite of what the app thinks its doing (e.g, if the user already pressed Mute manually).

Nothing in current pipeline to get this functionality in JTAPI API.