logo
down
shadow

Split a large csv file into multiple csv files according to the size in powershell


Split a large csv file into multiple csv files according to the size in powershell

By : jacob
Date : January 11 2021, 05:14 PM
around this issue this takes a slightly different path to a solution. [grin]
it ...
code :
$SourceDir = $env:TEMP
$InFileName = 'LargeFile.csv'
$InFullFileName = Join-Path -Path $SourceDir -ChildPath $InFileName

$BatchCount = 4

$DestDir = $env:TEMP
$OutFileName = 'LF_Batch_.csv'
$OutFullFileName = Join-Path -Path $DestDir -ChildPath $OutFileName

#region >>> build file to work with
#    remove this region when you are ready to do this with your test data OR to do this with real data
if (-not (Test-Path -LiteralPath $InFullFileName))
    {
    Get-ChildItem -LiteralPath $env:APPDATA -Recurse -File |
        Sort-Object -Property Name |
        Select-Object Name, Length, LastWriteTime, Directory |
        Export-Csv -LiteralPath $InFullFileName -NoTypeInformation
    }
#endregion >>> build file to work with

$CsvAsText = Get-Content -LiteralPath $InFullFileName
[array]$HeaderLine = $CsvAsText[0]
$BatchSize = [int]($CsvAsText.Count / $BatchCount) + 1

$StartLine = 1

foreach ($B_Index in 1..$BatchCount)
    {
    if ($B_Index -ne 1)
        {
        $StartLine = $StartLine + $BatchSize + 1
        }
    $CurrentOutFullFileName = $OutFullFileName.Replace('_.', ('_{0}.' -f $B_Index))
    $HeaderLine + $CsvAsText[$StartLine..($StartLine + $BatchSize)] |
        Set-Content -LiteralPath $CurrentOutFullFileName
    }


Share : facebook icon twitter icon
Windows Powershell: can I split a large SQL file into small files containing one db object

Windows Powershell: can I split a large SQL file into small files containing one db object


By : Lord Nermal
Date : March 29 2020, 07:55 AM
hope this fix your issue Here's a script that will split your large file into smaller files, starting a new file after each line that contains a semi-colon. It generates the filename based on the first non-blank line following the semicolon (in your example, the two filenames would be CREATE_SET_TABLE_CATALOG.SQL and CREATE_SET_TABLE_CHASSIS.SQL.
code :
$buffer = @()
$filename = $null
Get-Content '.\LARGEFILE.SQL' | ForEach-Object {
    $line = $_
    $buffer += @($line)

    if ([String]::IsNullOrWhiteSpace($filename)) {
        $filename = $line.Trim().Replace(',','') -replace '[^a-zA-Z0-9_.]+', '_'
    }

    if ($line.Contains(';')) {
        $buffer | Out-File "$filename.SQL" -Encoding utf8
        $buffer = @()
        $filename = $null
    }
}
Split a sql file into multiple files using powershell

Split a sql file into multiple files using powershell


By : Sandeep
Date : March 29 2020, 07:55 AM
wish of those help Hi I have a single file with multiple(~400) Insert statements like this and I need to split this to multiple files having one insert statement in each of those files. I am guessing it can be done using powershell effectively. , This should help get you going in the right direction:
code :
$desktop = [Environment]::GetFolderPath("Desktop") #saving to desktop, adjust as needed
$file = Get-Content "C:\Temp\test.sql" #replace w/ your file location
$content = "" #variable to hold content for each individual file

$pre = "Test_"
$key = ""
$tableName = ""

foreach ($line in $file)
{

    if($line -eq "") #if line is blank, export file and reset everything
    {
      $outFile = $desktop + "\" + $pre + $key + "_" + $tableName + ".sql"
      $content | Out-File $outFile

      $key = ""
      $tableName = ""
      $content = ""
    }
    else 
    { #otherwise, collect line
        $content += $line + "`r`n"
        $words = $line -split " "

        if($words[0] -eq "insert") #if this is the insert, take the table name
        {
          $tableName = $words[2]
        }

        if($words[0] -eq "values") #if this is the values, take the key
        {
          $k = $words[1] -split ","
          $key = $k[0].Replace("(","").Replace(",","")
        }
    }


} # end loop
# get last chunk for final file
if($content -ne "")
{
    $outFile = $desktop + "\" + $pre + $key + "_" + $tableName + ".sql"
    $content | Out-File $outFile
}
How can I split a large CSV file into multiple files of rougly equal size using bash tools alone?

How can I split a large CSV file into multiple files of rougly equal size using bash tools alone?


By : Levi
Date : March 29 2020, 07:55 AM
I wish this help you Here's one in awk. You provide it with filename and the maximum number of "lines" (for example -v m=3) you want in one file and it splits the file (based on your data) on lines that do not start with < so basically the header and the product title lines:
code :
$ awk -v m=3 'NR==1{j=0}{if($0!~/^</){i++;if(i>m){i=1;j++}};print > "split-" j}' file
$ ls -1rt
split-3
split-2
split-1
split-0
$ cat split-3
Product title 9,Sample,"<div class=""productdetails"">
<h2 style=""margin: 0px 0px 15px; line-height: 1.2; text-align: center;"">Title</h2>
<p style=""color: #333333; margin: 0px; font-size: 13px; line-height: 23.1111px; padding: 0px; font-family: sans-serif, Arial, Verdana, 'Trebuchet MS';""><strong>Features:</strong></p>
<ul style=""padding: 0px 40px; margin: 0px; color: #333333; font-family: sans-serif, Arial, Verdana, 'Trebuchet MS'; font-size: 13px; line-height: 20.8px;"">
<li style=""list-style: none;"">Testing testing</li>
<li style=""list-style: none;"">One two three</li>
</ul>
</div>"
awk -v m=3 '           # provide m
NR==1 {                # on the first record
    j=0                # set j to 0
}
{
    if($0!~/^</) {     # when a line not starting with a < is met
        i++            # increase line counter
        if(i>m) {      # if line counter exceeds max
            i=1        # reset it back to 1
            j++        # split file name index
        }
    }
    print > "split-" j # output
}' file
Split large excel file to multiple smaller file by user defined rows through powershell

Split large excel file to multiple smaller file by user defined rows through powershell


By : user1666581
Date : March 29 2020, 07:55 AM
To fix this issue You can utilize an awesome module developed by Doug finke . Import-Excel
below code will solve you problem.
code :
$r=@()
$t=$C=1
Import-Excel -Path C:\Temp\test.xlsx|Foreach-Object -Process {
    #Append rows in an array
    $r += $_

    #Save in a new excel when count reaches 3
    if($C -eq 3){
        $r | Export-Excel -Path C:\Temp\test_$t.xlsx

        #reset values
        $r=@()
        $c=1
       $t++
    }
    else{
        #increment row count
        $c++
    }
}

#save remaining rows
$r|Export-Excel -Path C:\Temp\test_$t.xlsx
How to split war files if the file size is large and deploy it in same context root

How to split war files if the file size is large and deploy it in same context root


By : Justin Zhang
Date : March 29 2020, 07:55 AM
seems to work fine I guess the majority of your WAR file size are libraries under /WEB-INF/lib. You can place most (if not all) JARs in JBoss directly, which will result in very small WAR. Take a look here: Where to put a shared library in JBoss AS 5?
Related Posts Related Posts :
  • Return file path and context text line after recursive search
  • How to compare integer literals?
  • How to set encoding for a herestring/heredoc in powershell?
  • How to Uninstall-Module of older version
  • How to ignore letters/words?
  • How to convert value to KB, MB, or GB depending on digit placeholders?
  • How to copy files based on last modified date to network drive?
  • Remove-PSDrive doesn't remove Mapped Drives
  • Wait for Active Directory Authentication URL list to update within a Powershell Azure Function
  • Trying to move AD user and disable the account running some PowerShell
  • How to run a local powershell script into remote machine via ansible?
  • Set ACL System.Security.AccessControl.FileSystemAccessRule to multiple users?
  • Replace Part of Filename if Filename Already Exists
  • Using a function in a calculated field in Powershell?
  • How to download a file accepting license using powershell
  • Using Powershell, how to add a task sequence for installing an SCCM package
  • Powershell Azure function times out on processing queue
  • How to get a last 15 minutes of a formatted log using a powershell script
  • Using both desktop Powershell 5.1 and Powershell Core 6.1
  • What does mean % , $_ and @ in Powershell?
  • Teams: Invoke-Webrequest sending base64 string (png) to Teams
  • How can I replicate New-SmbGlobalMapping in C# code?
  • How do I output the files being copied in console?
  • Powershell: How to stop start-job?
  • Formatting output from PowerShell to a csv file
  • How to check if a PowerShell switch parameter is absent or false
  • How do I force a function to return a single element array instead of the contained object?
  • compressed file syntax can be supported across all powershell version
  • Is there a way in Powershell for a boolean parameter to automatically be true if not specified when the function is call
  • New-Partition -DriveLetter Option
  • How to add SAMAccountName to output?
  • How to do looping to rename and read a file using PowerShell?
  • Read CSV file then split the values and create a global variable
  • (Get-Date).ToUniversalTime() convert to string but not change format
  • Adding Extra Headers in CSV
  • New-Mailbox command does not accept the -Equipment argument
  • How do I remove IIS 'server' header from http response using Powershell?
  • Creating Subdirectory With Date As Today(mm-dd-yyyy) in Network Drive
  • My first parameter is not being passed to my function
  • How can I overwrite an existing zip file?
  • How to use double if statement using powershell?
  • Command using either or both -Path parameter and $input?
  • How can I return multiple values from a powershell script to the batch file that is calling it?
  • Sensitive word search with powershell
  • Copy files located in multiple subfolder, using a mildmatch search
  • Replace the text for all files in a Directory
  • How to write a PowerShell script which auto increments a number by 1 every time it's run?
  • new user powershell get-service
  • Removing Parts of a File Name based on a Delimiter
  • How can we transform Date in CSV file using PowerShell scripts
  • How to replace multiple characters (<3) of all the filenames in a folder (and subfolder) using PowerShell
  • Powershell environment variable
  • Removing spaces from a string using Powershell
  • List Filename and current date of all files in a directory
  • PowerShell calling environment variable with another variable
  • Uninstall all software starting with a specific string
  • Set-ADUser does not handle empty or null variables
  • How do I get the data results from Invoke-RestMethod into CSV file?
  • ConvertFrom-String returns symbol not text
  • Get-WinEvent Select time and field within data item of 1
  • shadow
    Privacy Policy - Terms - Contact Us © festivalmusicasacra.org