Deleting a Task means that it will be deleted forever. There is no undo.
Closing a Task means that you still keep everything to do with the Task but it just moves to a Closed state.
You can't delete a Task if it has activity on it. For instance, if you have Time logged against a Task then you can't delete it. You can however close it.
Locate the Task that you wish delete and open up the Task sidebar
So long as there isn't any activity on the Task you will see a Delete button with a trash icon. This doesn't appear if there is activity on the Task meaning you can't delete the Task.
Click on this Delete button and the Task will be deleted immediately. Be careful as this cannot be undone.