Google has said time and time again that 404s are normal, they don't cause penalties or indexing issues. But a similar question came up about 410s......
-
Google: 410 Response Codes Don't Generally Cause Indexing Issues
seroundtable.com