본문 바로가기

Machine Learning/SageMaker

Amazon SageMaker & ML(sklearn)

이번 글은 SageMaker를 활용하여 기본적인 데이터 시각화 분석과 결과 평가 방법부터, 캐글의 Bike Rental 데이터를 통해 데이터 전처리, 트레이닝, 모델 생성, 배포 일련의 작업을 다룬다. 기본 sklearn을 사용해보고 SageMaker의 high level interface의 estimators 사용 방법과 비교한다.

 

개인적으로 학습한 내용이라 두서 없다. 시간은 업무와 병행했기 때문에 약 4주 정도 소요됐다. 나는 모두의딥러닝만 두어차례 본 초딩 수준임. 아무튼 고...

 

1. Introduce

1-1. np, pd, plt(plt.hist/블록, plt.scatter/분산, plt.plot/선형)

plt.hist(df.petal_length)

 

plt.scatter(df[setosa].index, df[setosa].petal_length, label ='set')

 

plt.plot(df['Vehicles'], label='target')

1-2. 데이터 전처리

vehicle = df['Vehicles'].fillna(0)

plt.plot(vehicles, ls='-.', alpha=0.8, label='mean')

fillna(0), fillna(df['Car'].mean()) / df['Car'].interpolate() / fillna(method='ffill') / fillna(method='bfill')

df['Vehicles'].fillna(0)

 

df['Vehicles'].fillna(df['Vehicles'].mean())

 

df['Vehicles'].interpolate()

 

df['Vehicles'].fillna(method='ffill')

 

df['Vehicles'].fillna(method='bfill')

1-3. 비선형 함수

plt.plot(df['exponential']), plt.plot(df['log']), plt.plot(np.arange(-20,20,.1), df['sine'] 등

plt.plot(df['exponential'])

 

2. PerformanceEvaluation

2-1. linear Regression

plots, RMSE, Residual Histograms(Over/Under)

 

2-2. binary_classfier_performance

Confusion metric

classification_report(df['Pass'], df[model.replace(' ','') + '_Prediction'], labels=[1,0], target_names=['Pass','Fail']))

https://www.youtube.com/watch?v=nMAtFhamoRY

2-3. binary_classifier_rawscore_evaluation(ROC & AUC)

roc_score = roc_auc_score(df['Pass'], df[model.replace(' ','') + '_Prediction'])

https://www.youtube.com/watch?v=4jRBRDbJemM

참고

 

Data Science School

Data Science School is an open space!

datascienceschool.net

 

2-4. multicall_classifier_performance

Accuracy(전체 적중률 TP+TN/ALL), Percision(예상에서 적중률 TP/TP+FP), Recall(실제에서 적중률 TP/TP+FN),

F1-score(per/rec 값을 이용한 스코어), Support(normalize)

Micro avg: ? / Macro avg: 평균 / Weighted avg: 가중 평균

classification_report( df['NumericClass'], df[model.replace(' ','') + '_Prediction'], labels=labels, target_names=classes)

3. SageMaker Overview

3-1. DataFormat of SageMaker

Training Data Format

CSV, RecordIO, Algorithm specific formats(LibSVM, JSON, Pqrquet) are stored in S3

 

Inference Format

CSV, JSON, RecordIO

import sagemaker.amazon.common as smac #RecordIO

import sagemaker.amazon.common as smac

def write_recordio_file (filename, x, y=None):
    with open(filename, 'wb') as f:
        smac.write_numpy_to_dense_tensor(f, x, y)
        
def read_recordio_file (filename, recordsToPrint = 10):
    with open(filename, 'rb') as f:
        record = smac.read_records(f)
        for i, r in enumerate(record):
            if i >= recordsToPrint:
                break
            print ("record: {}".format(i))
            print(r)

3-2. Mode

File, Pipe Mode

 

4. XGBoosts(Gradient Boosted Tress)

XGBoosts: 트리 형식의 가중치 알고리즘(Support Regression, Classification)

 

4-1. PerformanceEvaluation(Quadratic Regression, XGBoost VS Linear Regression)

- quadratic_xgboost_localmode.ipynb

- XGBoost

X_train = df_train.iloc[:,1:] # Features: 1st column onwards 
y_train = df_train.iloc[:,0].ravel() # Target: 0th column

X_validation = df_validation.iloc[:,1:]
y_validation = df_validation.iloc[:,0].ravel()

X_train
y_train
X_validation
y_validation

# max_depth = 5,objective="reg:linear",num_round = 50
# XGBoost Training Parameter Reference: 
#   https://github.com/dmlc/xgboost/blob/master/doc/parameter.md
regressor = xgb.XGBRegressor()
regressor
regressor.fit(X_train,y_train, eval_set = [(X_train, y_train), (X_validation, y_validation)])

[98] validation_0-rmse:1.49427 validation_1-rmse:3.71814

[99] validation_0-rmse:1.49156 validation_1-rmse:3.71637

 

# Get the Training RMSE and Evaluation RMSE
eval_result = regressor.evals_result()
eval_result

 

regressor.evals_result()

 

eval_result = regressor.evals_result()
training_rounds = range(len(eval_result['validation_0']['rmse']))
print(training_rounds)

plt.scatter(x=training_rounds,y=eval_result['validation_0']['rmse'],label='Training Error')
plt.scatter(x=training_rounds,y=eval_result['validation_1']['rmse'],label='Validation Error')
plt.grid(True)
plt.xlabel('Iteration')
plt.ylabel('RMSE')
plt.title('Training Vs Validation Error')
plt.legend()

regressor.evals_result()

 

xgb.plot_importance(regressor)
plt.show()

xgb.plot_importance(regressor)

참고

 

Stats Overflow

A Data Scientist's Tech Blog

aldente0630.github.io

 

- Validation Dateset Compare Actual and Predicted(quadratic 파일 Import가 빠져있음... 다시해야됨)

XGBoost!

#def quad_func (x):
#    return 5*x**2 -23*x + 47

df = pd.read_csv('quadratic_all.csv')
plt.scatter(df.x,df.y,label='Target')
plt.grid(True)
plt.xlabel('Input Feature')
plt.ylabel('Target')
plt.legend()
plt.title('Simple Regression Dataset')
plt.show()

Quadratic(5*x**2 - 23*x +47)

plt.title('XGBoost - Validation Dataset')
plt.scatter(df_validation.x,df_validation.y,label='actual',marker='.')
plt.scatter(df_validation.x,result,label='predicted',marker='.')
plt.grid(True)
plt.legend()
plt.show()

# RMSE Metrics
print('XGBoost Algorithm Metrics')
mse = mean_squared_error(df_validation.y,result)
print(" Mean Squared Error: {0:.2f}".format(mse))
print(" Root Mean Square Error: {0:.2f}".format(mse**.5))

XGboost RMSE

# Residual
# Over prediction and Under Prediction needs to be balanced
# Training Data Residuals
residuals = df_validation.y - result
plt.hist(residuals)
plt.grid(True)
plt.xlabel('Actual - Predicted')
plt.ylabel('Count')
plt.title('XGBoost Residual')
plt.axvline(color='r')
plt.show()

# Count number of values greater than zero and less than zero
value_counts = (residuals > 0).value_counts(sort=False)

print(' Under Estimation: {0}'.format(value_counts[True]))
print(' Over  Estimation: {0}'.format(value_counts[False]))

XGboost U/O

# Plot for entire dataset
plt.scatter(df.x,df.y,label='Target')
plt.scatter(df.x,regressor.predict(df[['x']]) ,label='Predicted')
plt.grid(True)
plt.xlabel('Input Feature')
plt.ylabel('Target')
plt.legend()
plt.title('XGBoost')
plt.show()

XGBoost Predicted

 

linear_regressor!

lin_regressor = LinearRegression()
lin_regressor.fit(X_train,y_train)
lin_regressor.coef_ #Gradient -> array([-21.4367])
lin_regressor.intercept_ # X: Zero Y Value -> 740.362

## -21.43 * X + 740.36
plt.title('LinearRegression - Validation Dataset')
plt.scatter(df_validation.x,df_validation.y,label='actual',marker='.')
plt.scatter(df_validation.x,result,label='predicted',marker='.')
plt.grid(True)
plt.legend()
plt.show()

# RMSE Metrics
print('Linear Regression Metrics')
mse = mean_squared_error(df_validation.y,result)
print(" Mean Squared Error: {0:.2f}".format(mse))
print(" Root Mean Square Error: {0:.2f}".format(mse**.5))

LR RMSE

# Residual
# Over prediction and Under Prediction needs to be balanced
# Training Data Residuals
residuals = df_validation.y - result
plt.hist(residuals)
plt.grid(True)
plt.xlabel('Actual - Predicted')
plt.ylabel('Count')
plt.title('Linear Regression Residual')
plt.axvline(color='r')
plt.show()

# Count number of values greater than zero and less than zero
value_counts = (residuals > 0).value_counts(sort=False)

print(' Under Estimation: {0}'.format(value_counts[True]))
print(' Over  Estimation: {0}'.format(value_counts[False]))

LR U/O

# Plot for entire dataset
plt.scatter(df.x,df.y,label='Target')
plt.scatter(df.x,lin_regressor.predict(df[['x']]) ,label='Predicted')
plt.grid(True)
plt.xlabel('Input Feature')
plt.ylabel('Target')
plt.legend()
plt.title('LinearRegression')
plt.show()

LR Predicted

Summary: 당연히 Non-linear의 경우 XGBoost가 효과적이며, Linear Regression의 경우 언더피팅이 발생

또한 추가 테스트로 XGboost의 경우 상/하한선이 존재함을 확인

 

 

4-2. Step

- Data Preparation/Exploration

df.corr()
df.corr()['count']

df.corr()
피어슨 상관계수

- Training on instance: biketrain_data_preparation_rev1/3.ipynb, xgboost_localmode_rev1/3.ipynb

코드 업데이트가 빠르네..4주만에 또 바꼈네... 

# Example
# Converts to log1p(count)
# Print original count back using expm1
print('Test log and exp')
test_count = 100
print('original value', test_count)
x = np.log1p(test_count) # log (x+1)
print('log1p', x)
print('expm1', np.expm1(x)) # exp(x) - 1

흐음...

regressor = xgb.XGBRegressor(max_depth=5,eta=0.1,subsample=0.7,num_round=150)

 

 

dmlc/xgboost

Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more. Runs on single machine, Hadoop, Spark, Flink and DataFlow - dmlc/xgboost

github.com

 

XGBoost 하이퍼파라미터 - Amazon SageMaker

XGBoost 하이퍼파라미터 다음 표에는 Amazon SageMaker XGBoost 알고리즘에 필요하거나 가장 일반적으로 사용되는 하이퍼파라미터 하위 세트가 포함되어 있습니다. 이들은 사용자가 데이터로부터 모델 파라미터의 예측을 촉진하기 위해 설정하는 파라미터입니다. 먼저 반드시 설정해야 하는 필수 하이퍼파라미터가 알파벳 순으로 나열되어 있습니다. 그 다음에 설정할 수 있는 선택적 하이퍼파라미터가 알파벳 순으로 나열되어 있습니다. Amazon SageM

docs.aws.amazon.com

training_rounds = range(len(eval_result['validation_0']['rmse']))
plt.scatter(x=training_rounds,y=eval_result['validation_0']['rmse'],label='Training Error')
plt.scatter(x=training_rounds,y=eval_result['validation_1']['rmse'],label='Validation Error')
plt.grid(True)
plt.xlabel('Input Feature')
plt.ylabel('RMSE')
plt.title('Training Vs Validation Error')
plt.legend()

RMSE evaluation

 

xgb.plot_importance(regressor)
plt.show()

xgb.plot_importance(regressor)

# Negative Values are predicted
df['count_predicted'].describe()

df['count_predicted'].describe()

def adjust_count(x):
    if x < 0:
        return 0
    else:
        return x
        
df['count_predicted'] = df['count_predicted'].map(adjust_count)
df[df['count_predicted'] < 0]
plt.boxplot([df['count'],df['count_predicted']], labels=['actual','predicted'])
plt.title('Box Plot - Actual, Predicted')
plt.ylabel('Target')
plt.grid(True)

plt.boxplot()

# Over prediction and Under Prediction needs to be balanced
# Training Data Residuals
residuals = (df['count_predicted'] - df['count'])

plt.hist(residuals)
plt.grid(True)
plt.xlabel('(Predicted - Actual)')
plt.ylabel('Count')
plt.title('Residuals Distribution')
plt.axvline(color='g')

plt.hist(residuals)

import sklearn.metrics as metrics
print("RMSE: {0}".format(metrics.mean_squared_error(df['count'],df['count_predicted'])**.5))

# Metric Use By Kaggle
def compute_rmsle(y_true, y_pred):
    if type(y_true) != np.ndarray:
        y_true = np.array(y_true)
        
    if type(y_pred) != np.ndarray:
        y_pred = np.array(y_pred)
     
    return(np.average((np.log1p(y_pred) - np.log1p(y_true))**2)**.5)
    
print("RMSLE: {0}".format(compute_rmsle(df['count'],df['count_predicted'])))

RMSLE Metric by Kaggle

 

Kaggle TEST 파일 Predict 후에 결과 값을 정제하고 Kaggle에 업로드

df_test[df_test["count"] < 0]
df_test["count"] = df_test["count"].map(adjust_count)
df_test[['datetime','count']].to_csv('predicted_count.csv',index=False)

Kaggle Score

엇... 수정된 코드로 스코어가 0.62 -> 0.40 포인트까지 줄었다.

핵심은 데이터정제 시 Count에 log1p를 사용

 

- Training on SageMaker: xgboost_cloud_training_template.ipynb

 

[Upload Data to S3]

# Specify your bucket name
bucket_name = 'leedoing-ml-sagemaker'

training_folder = 'bikerental/training/'
validation_folder = 'bikerental/validation/'
test_folder = 'bikerental/test/'

s3_model_output_location = 's3://{0}/bikerental/model'.format(bucket_name)
s3_training_file_location = 's3://{0}/{1}'.format(bucket_name,training_folder)
s3_validation_file_location = 's3://{0}/{1}'.format(bucket_name,validation_folder)
s3_test_file_location = 's3://{0}/{1}'.format(bucket_name,test_folder)

def write_to_s3(filename, bucket, key):
    with open(filename,'rb') as f: # Read in binary mode
        return boto3.Session().resource('s3').Bucket(bucket).Object(key).upload_fileobj(f)
        
write_to_s3('bike_train.csv', 
            bucket_name,
            training_folder + 'bike_train.csv')

write_to_s3('bike_validation.csv',
            bucket_name,
            validation_folder + 'bike_validation.csv')

write_to_s3('bike_test.csv',
            bucket_name,
            test_folder + 'bike_test.csv')

SageMaker 사용을 위해서 각 정제된 데이터를 S3에 저장

 

In the past, we had to maintain the algorithm containers mapping:


containers = {'us-west-2': '433757028032.dkr.ecr.us-west-2.amazonaws.com/xgboost:latest',
              'us-east-1': '811284229777.dkr.ecr.us-east-1.amazonaws.com/xgboost:latest',
              'us-east-2': '825641698319.dkr.ecr.us-east-2.amazonaws.com/xgboost:latest',
              'eu-west-1': '685385470294.dkr.ecr.eu-west-1.amazonaws.com/xgboost:latest'}

Reference:
https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-algo-docker-registry-paths.html

과거 ecr을 이용한 containers 정의가 필요했으나, 이제 그럴 필요가 없음.

 

[Training Algorithm Docker Image]

# Establish a session with AWS
sess = sagemaker.Session()
role = get_execution_role()

# Sagemaker API now maintains the algorithm container mapping for us
# Specify the region, algorithm and version
container = sagemaker.amazon.amazon_estimator.get_image_uri(
    sess.boto_region_name,
    "xgboost", 
    "latest")

print('Using SageMaker XGBoost container:\n{} ({})'.format(container, sess.boto_region_name))

ecr 생성 로그 확인

[Build Model]

# Configure the training job
# Specify type and number of instances to use
# S3 location where final artifacts needs to be stored

#   Reference: http://sagemaker.readthedocs.io/en/latest/estimators.html

estimator = sagemaker.estimator.Estimator(
    container,
    role, 
    train_instance_count=1, 
    train_instance_type='ml.m4.xlarge',
    output_path=s3_model_output_location,
    sagemaker_session=sess,
    base_job_name ='xgboost-bikerental-v1')

SageMaker의 high level interface의 estimators

 

# Specify hyper parameters that appropriate for the training algorithm
# XGBoost Training Parameter Reference: 
#   https://github.com/dmlc/xgboost/blob/master/doc/parameter.md

# max_depth=5,eta=0.1,subsample=0.7,num_round=150
estimator.set_hyperparameters(max_depth=5,
                              objective="reg:linear",
                              eta=0.1,
                              num_round=150)

estimator.hyperparameters()

hyperparameters

Hyperparameters 구성

 

XGBoost 하이퍼파라미터 - Amazon SageMaker

XGBoost 하이퍼파라미터 다음 표에는 Amazon SageMaker XGBoost 알고리즘에 필요하거나 가장 일반적으로 사용되는 하이퍼파라미터 하위 세트가 포함되어 있습니다. 이들은 사용자가 데이터로부터 모델 파라미터의 예측을 촉진하기 위해 설정하는 파라미터입니다. 먼저 반드시 설정해야 하는 필수 하이퍼파라미터가 알파벳 순으로 나열되어 있습니다. 그 다음에 설정할 수 있는 선택적 하이퍼파라미터가 알파벳 순으로 나열되어 있습니다. Amazon SageM

docs.aws.amazon.com

 

[Specify Training Data Location and Optionally, Validation Data Locaion]

# content type can be libsvm or csv for XGBoost
training_input_config = sagemaker.session.s3_input(
    s3_data=s3_training_file_location,
    content_type='csv',
    s3_data_type='S3Prefix')

validation_input_config = sagemaker.session.s3_input(
    s3_data=s3_validation_file_location,
    content_type='csv',
    s3_data_type='S3Prefix'
)

data_channels = {'train': training_input_config, 'validation': validation_input_config}
print(training_input_config.config)
print(validation_input_config.config)

[Train the model]

# XGBoost supports "train", "validation" channels
# Reference: Supported channels by algorithm
#   https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-algo-docker-registry-paths.html
estimator.fit(data_channels)

estimator.fit(data_channels)

 

모델 트레이닝은 약 3이 소요된다. 물론 데이터에 따라 다름...

Completed 메시지와 아래와 같이 쥬피터에서 RMSE 결과를 확인

estimator.fit 결과

[Deploy Model]

# Ref: http://sagemaker.readthedocs.io/en/latest/estimators.html
predictor = estimator.deploy(initial_instance_count=1,
                             instance_type='ml.m4.xlarge',
                             endpoint_name = 'xgboost-bikerental-v1')

estimator.deploy를 통해 모델을 배포한다.

m4.xlarge로 돈을 뿌리며 배포 중..

배포는 약 5분 정도 소요된다.

 

[Predictions]

from sagemaker.predictor import csv_serializer, json_deserializer

predictor.content_type = 'text/csv'
predictor.serializer = csv_serializer
predictor.deserializer = None
predictor.predict([[3,0,1,2,28.7,33.335,79,12.998,2011,7,7,3]])

 

SageMaker와 XGBoost 사용 정리

1. Upload Train and Validation file to S3

2. Specify Algorithm and Hyperparameters

3. Configure type of server and number of servers to user for Training

4. Create a real-time Endpoint for interactive use case

 

- Prediction using SageMaker: xgboost_cloud_prediction_template.ipynb

1. Invoke Endpoint for interactive use cases

2. Connect to an exisiting Endpoint

3. Endpoint Security

4. Multiple obserations in a single round trip

import boto3
import re
from sagemaker import get_execution_role
import sagemaker

# Acquire a realtime endpoint
endpoint_name = 'xgboost-bikerental-v1'
predictor = sagemaker.predictor.RealTimePredictor(endpoint=endpoint_name)

from sagemaker.predictor import csv_serializer, json_deserializer

predictor.content_type = 'text/csv'
predictor.serializer = csv_serializer
predictor.deserializer = None
df_all = pd.read_csv('bike_test.csv')
# Need to pass an array to the prediction
# can pass a numpy array or a list of values [[19,1],[20,1]]
arr_test = df_all[df_all.columns[1:]].values ##date를 빼고 np 형태로 변경
type(arr_test)
arr_test.shape
arr_test[:5]

numpy.ndarray

 

result = predictor.predict(arr_test[:2])
result

predict(arr_test[:2])

 

TEST 배열을 나눠서 predict

# For large number of predictions, we can split the input data and
# Query the prediction service.
# array_split is convenient to specify how many splits are needed
predictions = []
for arr in np.array_split(arr_test,10):
    result = predictor.predict(arr)
    result = result.decode("utf-8")
    result = result.split(',')
    print (arr.shape)
    predictions += [float(r) for r in result]
np.expm1(predictions)
df_all['count'] = np.expm1(predictions) ##Count reverse(np.log1p)

np.expm1(predictions)

 

 

numpy.expm1 — NumPy v1.17 Manual

Parameters: x : array_like Input values. out : ndarray, None, or tuple of ndarray and None, optional A location into which the result is stored. If provided, it must have a shape that the inputs broadcast to. If not provided or None, a freshly-allocated ar

docs.scipy.org

df_all.head()
df_all[['datetime','count']].to_csv('predicted_count_cloud.csv', index=False)

BikeRental 캐글

Scaling based on Invocations

MAX RPS: 100

Safety Factor = 0.5

Target SageMakerVariantInvocationsPerInstance = (MAX_RPS * SAFETY_FACTOR) * 60

SageMakerVariantInvocationsPerInstance = 100 * 0.5 * 60 = 3,000

 

일단 데이터의 기본부터 캐글 데이터를 사용하여 SageMaker의 XGBoost 활용까지 정리해봤다. 대충 한 바퀴 돌려봤는데 꽤 내용이 길어졌네. 여기서 한 번 자르고 간다.

'Machine Learning > SageMaker' 카테고리의 다른 글

Amazon SageMaker & ML(sklearn)  (0) 2019.09.26
Amazon SageMaker 샘플 비교  (0) 2019.02.15