compare records

Ok, I am looking for some advice on how to approach this work task. I have a text file containing lines of records. I am looking for a good/simple way to evaluate this file and write to separate file any duplicate records and any records with null values in the last field. My first thought was to read the records into an array....but then what would/should I do?...make a copy of the array then compare the 2 arrays against eachother?


here is a sample of the data:(the actual txt file would contain between 20,000 - 30,000 records)

111456,1,NGT FAMILY PRACTICE LPN/MA
700002,1,NGT FAMILY PRACTICE PCR
700002,2,
AASEDM1,1,OLY ALLERGY LPN/MA


any advice you could offer will be appreciated

I would load them in an array and sort the array.
After you have sorted the array you can loop through it once and compare the current line with the previous line, if the current line is the same as the previous write it to the output file.