Credit notes

I am self-employed in secondary occupation and do have a vAT number, but I am exempted from VAT. From September to December, I sent my invoices to my customer with VAT, as I thought I was subject to VAT. This is what I was told by the social secretariat. It was only when I submitted my quarterly return that I realised I am exempted because my turnover is less than £25,000 per year.

I would therefore like to cancel the previous invoices to my customer. I create a credit note via Dexxter, but I notice that the amount on the credit note is positive. Is this correct? I want to cancel the previous invoice and then resend an invoice without VAT and refund the difference (vAT) to the customer.