Not quite a mapreduce implementation :-D , but nevertheless …

while read l; do echo "SELECT id, email FROM tblName WHERE email='$l' ORDER by id DESC LIMIT 1;"|mysql -u root dbName; done < /tmp/dupes3.txt |grep -v 'email'|cut -f1|sed 's/^\(.*\)$/DELETE FROM tblName WHERE id=\1;/g' > /tmp/deletedupes3.sql

By shashi

2 thoughts on “Crazy shell scripting”
  1. Good one 🙂

    As per my understanding,
    1. dupes.txt with hundreds of records,
    2. Split that file into dupes1.txt, dupes2.txt dupesN.txt
    3. run above script in multiple system
    4. Combine the files generated by each system and run to remove duplicates in DB.

    but how the dupes3.txt(or dupes.txt) populated?
    echo SELECT email from tblName group by email having count(email) > 1 | mysql -uroot dbName | grep -v email > dupes.txt

    correct me if i am wrong.

Leave a Reply

Your email address will not be published. Required fields are marked *