Cheat Sheet for Data Manipulation with Python for Machine Learning and Data Science

File Input / Output

Reading CSV File into numpy array # Reading CSV File import csv import numpy as np vals = [] with open("fname.csv", 'r') as csvfile: data = csv.reader(csvfile, delimiter=',') for row in data: vals.append(row) # removing title row vals = vals[1:] y = np.array(vals).astype(np.float) Writing to CSV File

will create new file if does not exist, otherwise will append import csv import os def save (fn, row): if (os.path.isfile(fn)): m="a" else: m="w" with open(fn, m, encoding="utf8", newline='' ) as csvfile: fieldnames = ['f1', 'f2'] writer = csv.DictWriter(csvfile, fieldnames=fieldnames) if (m=="w"): writer.writeheader() writer.writerow(row) save ("fname.csv",{'f1' : 'this is f1', 'f2': 'this is f2'}) Writing to txt File
use a for append, w - write, wb - write byte-like object with open("fname.txt", 'w') as out: out.write("data" + '\n') Input to Pandas from CSV File
Additionally select needed columns, rows from dataframe import pandas as pd dt = pd.read_csv ("fname.csv") print (dt.columns.values) #delete columns 2,3 data=dt.drop(dt.columns[[2,3]], axis=1) #rename first two columns if needed data.columns = ['f1', 'f2'] #use only last 10 rows data_test = data[-10:] print (data_test)

python data processing for machine learning and data science

Pandas Dataframe and Array

How to navigate display, navigate dataframe. How to do one-hot encoding .
pandas.DataFrame is two-dimensional size-mutable tabular data structure with labeled axes
Displaying Pandas DataFrame import pandas as pd columns = ['0','1','2','3'] #create empty dataframe df = pd.DataFrame(columns = columns) pd.set_option('display.width', 1000) # Insert row at the end idx = len(df) df.loc[idx] = [1,8,3,1] # modify the value of one cell df.iloc[0,2] = 5 print(df) Array Initialization ar1dim = [0 for x in range(500)] ar2dim = [[0 for x in range(500)] for y in range(500)] ar3dim = [[[0 for x in range(500)] for y in range(500)] for z in range(500)] Working with Pandas # Reading to panda and converting categorical column to numerical (one-hot coding) # and keeping last column in the end import pandas as pd df = pd.read_csv("filename.csv", sep= ',' ) cols = df.columns.tolist() print (df) # get last column name last_col_header = cols[-1] df=pd.get_dummies(df) cols = df.columns.tolist() #move prev. last column to the end cols.insert(len(df.columns)-1, cols.pop(cols.index(last_col_header))) df = df.reindex(columns= cols) # navigating through dataframe for index, row in df.iterrows(): df.loc[index, "Y"] = df.loc[index, "Y"] + 5 print (df) """ prints in the beginning f1 f2 Y 0 11 ab 3 1 12 ab 2 2 14 afgh 7 prints in the end f1 f2_ab f2_afgh Y 0 11 1 0 8 1 12 1 0 7 2 14 0 1 12 """

JSON and API Data

Getting data from web with urllib and reading json data file and navigating through folder ## getting data from web with with urllib import os import csv import json url_link="http://hostname.com/link" from urllib.request import urlopen with urlopen(url_link) as url: data = url.read() #reading json file with open("jsonfile.txt") as json_file: json_data = json.load(json_file) #navigating through folders #opening and reading csv files import os import csv ## utf8 was used to support international characters ## path is the path folders where CSV files with the data are located path="C:\\Users\\Owner\\Downloads" list_of_files = {} for filename in os.listdir(path): # if the element is an CSV file then... if filename[-4:] == ".csv": list_of_files[filename] = path + "\\" + filename with open(list_of_files[filename], encoding="utf8" ) as f: csv_f = csv.reader(f) for i, row in enumerate(csv_f): if len(row) > 1 : #process data Getting json data from web import requests resp=requests.get("http://api_json_data") if resp.status_code != 200: print ("Something is wrong") exit(); # 'results' and 'key' are api specific names message=resp.json().get('results') for item in message: print (item['key']) Accessing the NumPy multidimensional array import numpy ar = numpy.array([[1, 2], [3, 4], [5, 6]]) print ( ar[:,0] ) # will print [1, 3, 5] print ( ar[1,:] ) # will print [3, 4]