Read Multiple CSV Files into one Frame in Python

Link to Source data

Link to Notebook

Pandas

Via read_csv

import pandas as pd
from pathlib import Path

# create a Path instance and filter for only csv files
files = Path("./Data_files/multiple_csvs/").rglob("*.csv")

# read in all the csv files
all_csvs = [pd.read_csv(file) for file in files]

# lump into one table
all_csvs = pd.concat(all_csvs)

all_csvs

Via read_csv and the command line

import subprocess
from io import StringIO

data = subprocess.run("awk '(NR==1) || (FNR>1)' ./Data_files/multiple_csvs/*.csv", 
                      shell=True, 
                      capture_output=True,
                      text=True).stdout

df = pd.read_csv(StringIO(data))

df

datatable

Via iread

from datatable import fread, iread, rbind

files = Path("./Data_files/multiple_csvs/").rglob("*.csv")

# iread returns an iterator of all files
all_csvs = iread(tuple(files))

# combine into one table with rbind
all_csvs = rbind(all_csvs)

all_csvs

Via fread

files = Path("./Data_files/multiple_csvs/").rglob("*.csv")

all_csvs = [fread(filename) for filename in files]

# combine into one frame
rbind(all_csvs)

Via fread and the command line

# less verbose than the Pandas option
all_csvs = fread(cmd="awk '(NR==1) || (FNR>1)' ./Data_files/multiple_csvs/*.csv")
all_csvs

Resources Used

datatable - fread

datatable - iread

pandas - read_csv

python - subprocess

python - pathlib

StackOverflow - read csv files via the command line