Selected annotations, groups and qualifiers can be deleted using the Delete key.
To remove an annotation object from the active view, select the object in the Annotations editor and press the Shift-Delete. Note that the object will not be removed from the project, but just from the active Sequence View. To add object again just drag and drop it to the Sequence View.