Python script to deduplicate lines in multiple files

» python » Python script to deduplicate lines in multiple files

By : user3100420

Date : January 12 2021, 07:00 PM

fixed the issue. Will look into that further yes, you should add the relative path (path to directory) when opening the file, as such

code :

cur_file = open(os.path.join(path, f), 'r')

output_file.write("\n".join(seen_lines))

for f in myfiles:
    cur_file = open(os.path.join(path, f), 'r')
    lines = set(cur_file.readlines())
    cur_file.close()
    with open(os.path.join(path, f), 'w') as of:
        of.write("\n".join(lines))

Share :

Perl script to search and replace multiple lines in multiple html files

By : Jalal

Date : March 29 2020, 07:55 AM

To fix this issue perl -0777 -i.withdiv -pe 's{]+?id="user-info"[^>]*>.*?

}{}gsmi;' test.html
-0777 means split on nothing, so slurp in whole file (instead of line by line, the default for -p

Shell script copying lines from multiple files

By : Goncho

Date : March 29 2020, 07:55 AM

wish of those help I believe ordering of files is also important to make sure you get output in desired sequence.
Consider this script:

code :

n=8
while read f; do
   sed $n'q;d' "$f" >> output.txt
   ((n+=8))
done < <(printf "%s\n" values_*.txt|sort -t_ -nk2,2)

Passing individual lines from files into a python script using a bash script

By : Rich

Date : March 29 2020, 07:55 AM

hop of those help? This might be a simple question, but I am new to bash scripting and have spent quite a bit of time on this with no luck; I hope I can get an answer here. , You probably meant:

code :

while read line; do
    VALUE=$line   ## No spaces allowed
    python fits_edit_head.py "$line" "$line" NEW_PARA 5  ## Quote properly to isolate arguments well
    echo "$VALUE+huh"  ## You don't expand without $
done < infile.txt

while read -u 4 line; do
     ...
done 4< infile.txt

readarray -t lines < infile.txt
for line in "${lines[@]}; do
    ...
done

Python script counting lines in multiple text files in one directory and producing simple report

By : nxdprinciple

Date : March 29 2020, 07:55 AM

wish of those help I need a python script counting lines in all text files in one directory and producing a general report on number of files with n number of lines. , Your names dict looks like that:

code :

{
    'file1.txt': 30,
    'file2.txt': 26,
    'file3.txt': 19,
    'file4.txt': 19
}

from collections import defaultdict

lines = defaultdict(int)
for val in names.values():
    lines[val] += 1

for k, v in lines.items():
    print("Files with {} lines: {}".format(k, v))

Files with 19 lines: 2
Files with 26 lines: 1
Files with 30 lines: 1

deduplicate records in multiple CSV files with varying columns

By : Sarvesh Kaushik

Date : March 29 2020, 07:55 AM

I think the issue was by ths following , I do not known how to make it simpler. I have an equal script done for some data of mine. It just runs twice, first to determine the min / max cols in all documents and finally to rewrite the csv files in an new folder, to keep the original data.
I am just using the csv lib from python. https://docs.python.org/2/library/csv.html

code :

import os
import csv

mincols = 0xffffffff
maxcols = 0

srcdir = '/tmp/csv/'
dstdir = '/tmp/csv2/'

for dirName, subdirList, fileList in os.walk(srcdir):
    for fname in fileList:
        if fname[-4:].lower() == '.csv':
            with open(os.path.join(dirName, fname)) as csvfile:
                reader = csv.reader(csvfile, delimiter=',', quotechar='"')
                for row in reader:
                    if mincols > len(row):
                        mincols = len(row)
                    if maxcols < len(row):
                        maxcols = len(row)

print(mincols, maxcols)

for dirName, subdirList, fileList in os.walk(srcdir):
    for fname in fileList:
        if fname[-4:].lower() == '.csv':
            fullpath = os.path.join(dirName, fname)    
            newfile = os.path.join(dstdir, fullpath[len(srcdir):])

            if not os.path.exists(os.path.dirname(newfile)):
                os.makedirs(os.path.dirname(newfile))
            with open(fullpath) as csvfile:
                reader = csv.reader(csvfile, delimiter=',', quotechar='"')
                with open(newfile, 'w') as dstfile:
                    writer = csv.writer(dstfile, delimiter=',', quotechar='"',
                        quoting=csv.QUOTE_MINIMAL)
                    for row in reader:
                        #You can deduplicate here 
                        writer.writerow(row[:mincols])