You can only delete a custom content block created by you using the following method above. Blocks created by modules can not be deleted using this method.

On the Block overview page (Admin > Structure > Block), blocks that can be deleted will have a "delete" link beside them. Clicking this link will present a confirmation page, and clicking the Delete button will remove the block.

Master Drupal in 7 hours - Basic Customization - Manage block - Delete block