Click on the file or folder and press the key combination shift+del together in order to remove the file or folder permanently.

Please note you will need the appropriate access permissions in order to do this.