Want to share your content on R-bloggers? click here if you have a blog, or here if you don’t.
We’ve always been told that decision trees are best for Gradient Boosting Machine Learning. I’ve always wanted to see for myself. AdaBoostClassifier is working well, but is relatively slow (by my own standards). A few days ago, I noticed that my Cython implementation of LSBoost in Python package mlsauce was already quite generic (never noticed before), and I decided to adapt it to any machine learning model with fit
and predict
methods. It’s worth mentioning that only regression algorithms are accepted as base learners, and classification is regression-based. The results are promising indeed; I’ll let you see for yourself below. All the algorithms, including xgboost
and RandomForest
, are used with their default hyperparameters. Which means, there’s still a room for improvement.
Install mlsauce (version 0.20.3) from GitHub:
!pip install git+https://github.com/Techtonique/mlsauce.git --verbose --upgrade --no-cache-dir import os import pandas as pd import mlsauce as ms from sklearn.datasets import load_breast_cancer, load_iris, load_wine, load_digits from sklearn.model_selection import train_test_split from time import time load_models = [load_breast_cancer, load_wine, load_iris] for model in load_models: data = model() X = data.data y= data.target X = pd.DataFrame(X, columns=data.feature_names) X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 13) clf = ms.LazyBoostingClassifier(verbose=0, ignore_warnings=True, custom_metric=None, preprocess=False) start = time() models, predictions = clf.fit(X_train, X_test, y_train, y_test) print(f"nElapsed: {time() - start} secondsn") display(models) 2it [00:01, 1.52it/s] 100%|██████████| 30/30 [00:21<00:00, 1.38it/s] Elapsed: 23.019137859344482 seconds
Accuracy | Balanced Accuracy | ROC AUC | F1 Score | Time Taken | |
---|---|---|---|---|---|
Model | |||||
GenericBooster(LinearRegression) | 0.99 | 0.99 | 0.99 | 0.99 | 0.35 |
GenericBooster(Ridge) | 0.99 | 0.99 | 0.99 | 0.99 | 0.27 |
GenericBooster(RidgeCV) | 0.99 | 0.99 | 0.99 | 0.99 | 1.07 |
GenericBooster(TransformedTargetRegressor) | 0.99 | 0.99 | 0.99 | 0.99 | 0.40 |
GenericBooster(KernelRidge) | 0.97 | 0.96 | 0.96 | 0.97 | 2.05 |
XGBClassifier | 0.96 | 0.96 | 0.96 | 0.96 | 0.91 |
GenericBooster(ExtraTreeRegressor) | 0.94 | 0.94 | 0.94 | 0.94 | 0.25 |
RandomForestClassifier | 0.92 | 0.93 | 0.93 | 0.92 | 0.40 |
GenericBooster(RANSACRegressor) | 0.90 | 0.86 | 0.86 | 0.90 | 15.22 |
GenericBooster(DecisionTreeRegressor) | 0.87 | 0.88 | 0.88 | 0.87 | 0.98 |
GenericBooster(KNeighborsRegressor) | 0.87 | 0.89 | 0.89 | 0.87 | 0.49 |
GenericBooster(ElasticNet) | 0.85 | 0.76 | 0.76 | 0.84 | 0.10 |
GenericBooster(Lasso) | 0.82 | 0.71 | 0.71 | 0.79 | 0.09 |
GenericBooster(LassoLars) | 0.82 | 0.71 | 0.71 | 0.79 | 0.10 |
GenericBooster(DummyRegressor) | 0.68 | 0.50 | 0.50 | 0.56 | 0.01 |
2it [00:00, 8.29it/s] 100%|██████████| 30/30 [00:15<00:00, 1.92it/s] Elapsed: 15.911818265914917 seconds
Accuracy | Balanced Accuracy | ROC AUC | F1 Score | Time Taken | |
---|---|---|---|---|---|
Model | |||||
RandomForestClassifier | 1.00 | 1.00 | None | 1.00 | 0.18 |
GenericBooster(ExtraTreeRegressor) | 1.00 | 1.00 | None | 1.00 | 0.16 |
GenericBooster(KernelRidge) | 1.00 | 1.00 | None | 1.00 | 0.38 |
GenericBooster(LinearRegression) | 1.00 | 1.00 | None | 1.00 | 0.23 |
GenericBooster(Ridge) | 1.00 | 1.00 | None | 1.00 | 0.17 |
GenericBooster(RidgeCV) | 1.00 | 1.00 | None | 1.00 | 0.24 |
GenericBooster(TransformedTargetRegressor) | 1.00 | 1.00 | None | 1.00 | 0.26 |
XGBClassifier | 0.97 | 0.96 | None | 0.97 | 0.06 |
GenericBooster(Lars) | 0.94 | 0.94 | None | 0.95 | 0.99 |
GenericBooster(DecisionTreeRegressor) | 0.92 | 0.92 | None | 0.92 | 0.23 |
GenericBooster(KNeighborsRegressor) | 0.92 | 0.93 | None | 0.92 | 0.21 |
GenericBooster(RANSACRegressor) | 0.81 | 0.81 | None | 0.80 | 12.63 |
GenericBooster(ElasticNet) | 0.61 | 0.53 | None | 0.53 | 0.04 |
GenericBooster(DummyRegressor) | 0.42 | 0.33 | None | 0.25 | 0.01 |
GenericBooster(Lasso) | 0.42 | 0.33 | None | 0.25 | 0.02 |
GenericBooster(LassoLars) | 0.42 | 0.33 | None | 0.25 | 0.01 |
2it [00:00, 5.14it/s] 100%|██████████| 30/30 [00:15<00:00, 1.92it/s] Elapsed: 16.0275661945343 seconds
Accuracy | Balanced Accuracy | ROC AUC | F1 Score | Time Taken | |
---|---|---|---|---|---|
Model | |||||
GenericBooster(Ridge) | 1.00 | 1.00 | None | 1.00 | 0.23 |
GenericBooster(RidgeCV) | 1.00 | 1.00 | None | 1.00 | 0.25 |
RandomForestClassifier | 0.97 | 0.97 | None | 0.97 | 0.26 |
XGBClassifier | 0.97 | 0.97 | None | 0.97 | 0.12 |
GenericBooster(DecisionTreeRegressor) | 0.97 | 0.97 | None | 0.97 | 0.27 |
GenericBooster(ExtraTreeRegressor) | 0.97 | 0.97 | None | 0.97 | 0.22 |
GenericBooster(LinearRegression) | 0.97 | 0.97 | None | 0.97 | 0.15 |
GenericBooster(TransformedTargetRegressor) | 0.97 | 0.97 | None | 0.97 | 0.37 |
GenericBooster(KNeighborsRegressor) | 0.93 | 0.95 | None | 0.93 | 1.52 |
GenericBooster(KernelRidge) | 0.87 | 0.83 | None | 0.85 | 0.63 |
GenericBooster(RANSACRegressor) | 0.63 | 0.59 | None | 0.61 | 10.86 |
GenericBooster(Lars) | 0.50 | 0.46 | None | 0.48 | 0.99 |
GenericBooster(DummyRegressor) | 0.27 | 0.33 | None | 0.11 | 0.01 |
GenericBooster(ElasticNet) | 0.27 | 0.33 | None | 0.11 | 0.01 |
GenericBooster(Lasso) | 0.27 | 0.33 | None | 0.11 | 0.01 |
GenericBooster(LassoLars) | 0.27 | 0.33 | None | 0.11 | 0.01 |
!pip install shap import shap best_model = clf.get_best_model() # load JS visualization code to notebook shap.initjs() # explain all the predictions in the test set explainer = shap.KernelExplainer(best_model.predict_proba, X_train) shap_values = explainer.shap_values(X_test) # this is multiclass so we only visualize the contributions to first class (hence index 0) shap.force_plot(explainer.expected_value[0], shap_values[..., 0], X_test)
“,e=e.removeChild(e.firstChild)):”string”==typeof r.is?e=l.createElement(n,{is:r.is}):(e=l.createElement(n),”select”===n&&(l=e,r.multiple?l.multiple=!0:r.size&&(l.size=r.size))):e=l.createElementNS(e,n),e[pa]=t,e[da]=r,Lu(e,t,!1,!1),t.stateNode=e;e:{switch(l=be(n,r),n){case”dialog”:Ur(“cancel”,e),Ur(“close”,e),a=r;break;case”iframe”:case”object”:case”embed”:Ur(“load”,e),a=r;break;case”video”:case”audio”:for(a=0;a53)return null;”w”in i||(i.w=1),”Z”in i?(a=(r=kr(Sr(i.y,0,1))).getUTCDay(),r=a>4||0===aur.ceil(r):ur(r),r=Gn.offset(r,7*(i.V-1)),i.y=r.getUTCFullYear(),i.m=r.getUTCMonth(),i.d=r.getUTCDate()+(i.w+6)%7):(a=(r=xr(Sr(i.y,0,1))).getDay(),r=a>4||0===aJn.ceil(r):Jn(r),r=Yn.offset(r,7*(i.V-1)),i.y=r.getFullYear(),i.m=r.getMonth(),i.d=r.getDate()+(i.w+6)%7)}else(“W”in i||”U”in i)&&(“w”in i||(i.w=”u”in i?i.u%7:”W”in i?1:0),a=”Z”in i?kr(Sr(i.y,0,1)).getUTCDay():xr(Sr(i.y,0,1)).getDay(),i.m=0,i.d=”W”in i?(i.w+6)%7+7*i.W-(a+5)%7:i.w+7*i.U-(a+6)%7);return”Z”in i?(i.H+=i.Z/100|0,i.M+=i.Z%100,kr(i)):xr(i)}}function S(e,t,n,r){for(var a,i,o=0,u=t.length,l=n.length;o=l)return-1;if(37===(a=t.charCodeAt(o++))){if(a=t.charAt(o++),!(i=w[a in Mr?t.charAt(o++):a])||(r=i(e,n,r))<0)return-1}else if(a!=n.charCodeAt(r++))return-1}return r}return b.x=x(n,b),b.X=x(r,b),b.c=x(t,b),_.x=x(n,_),_.X=x(r,_),_.c=x(t,_),{format:function(e){var t=x(e+="",b);return t.toString=function(){return e},t},parse:function(e){var t=k(e+="",!1);return t.toString=function(){return e},t},utcFormat:function(e){var t=x(e+="",_);return t.toString=function(){return e},t},utcParse:function(e){var t=k(e+="",!0);return t.toString=function(){return e},t}}}({dateTime:"%x, %X",date:"%-m/%-d/%Y",time:"%-I:%M:%S %p",periods:["AM","PM"],days:["Sunday","Monday","Tuesday","Wednesday","Thursday","Friday","Saturday"],shortDays:["Sun","Mon","Tue","Wed","Thu","Fri","Sat"],months:["January","February","March","April","May","June","July","August","September","October","November","December"],shortMonths:["Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"]}),Cr=Er.format,Tr=Er.parse,Er.utcFormat,Er.utcParse;var ni=function(t){!function(e,t){if("function"!=typeof t&&null!==t)throw new TypeError("Super expression must either be null or a function");e.prototype=Object.create(t&&t.prototype,{constructor:{value:e,writable:!0,configurable:!0}}),Object.defineProperty(e,"prototype",{writable:!1}),t&&Ja(e,t)}(u,t);var n,r,a,i,o=(a=u,i=function(){if("undefined"==typeof Reflect||!Reflect.construct)return!1;if(Reflect.construct.sham)return!1;if("function"==typeof Proxy)return!0;try{return Boolean.prototype.valueOf.call(Reflect.construct(Boolean,[],(function(){}))),!0}catch(e){return!1}}(),function(){var e,t=ti(a);if(i){var n=ti(this).constructor;e=Reflect.construct(t,arguments,n)}else e=t.apply(this,arguments);return function(e,t){if(t&&("object"===Za(t)||"function"==typeof t))return t;if(void 0!==t)throw new TypeError("Derived constructors may only return object or undefined");return ei(e)}(this,e)});function u(){var e;return function(e,t){if(!(e instanceof t))throw new TypeError("Cannot call a class as a function")}(this,u),e=o.call(this),window.lastAdditiveForceArrayVisualizer=ei(e),e.topOffset=28,e.leftOffset=80,e.height=350,e.effectFormat=ze(".2"),e.redraw=(0,Re.debounce)((function(){return e.draw()}),200),e}return n=u,(r=[{key:"componentDidMount",value:function(){var e=this;this.mainGroup=this.svg.append("g"),this.onTopGroup=this.svg.append("g"),this.xaxisElement=this.onTopGroup.append("g").attr("transform","translate(0,35)").attr("class","force-bar-array-xaxis"),this.yaxisElement=this.onTopGroup.append("g").attr("transform","translate(0,35)").attr("class","force-bar-array-yaxis"),this.hoverGroup1=this.svg.append("g"),this.hoverGroup2=this.svg.append("g"),this.baseValueTitle=this.svg.append("text"),this.hoverLine=this.svg.append("line"),this.hoverxOutline=this.svg.append("text").attr("text-anchor","middle").attr("font-weight","bold").attr("fill","#fff").attr("stroke","#fff").attr("stroke-width","6").attr("font-size","12px"),this.hoverx=this.svg.append("text").attr("text-anchor","middle").attr("font-weight","bold").attr("fill","#000").attr("font-size","12px"),this.hoverxTitle=this.svg.append("text").attr("text-anchor","middle").attr("opacity",.6).attr("font-size","12px"),this.hoveryOutline=this.svg.append("text").attr("text-anchor","end").attr("font-weight","bold").attr("fill","#fff").attr("stroke","#fff").attr("stroke-width","6").attr("font-size","12px"),this.hovery=this.svg.append("text").attr("text-anchor","end").attr("font-weight","bold").attr("fill","#000").attr("font-size","12px"),this.xlabel=this.wrapper.select(".additive-force-array-xlabel"),this.ylabel=this.wrapper.select(".additive-force-array-ylabel");var t=void 0;"string"==typeof this.props.plot_cmap?this.props.plot_cmap in je.colors?t=je.colors[this.props.plot_cmap]:(console.log("Invalid color map name, reverting to default."),t=je.colors.RdBu):Array.isArray(this.props.plot_cmap)&&(t=this.props.plot_cmap),this.colors=t.map((function(e){return q(e)})),this.brighterColors=[1.45,1.6].map((function(t,n){return e.colors[n].brighter(t)}));var n=ze(",.4");null!=this.props.ordering_keys&&null!=this.props.ordering_keys_time_format?(this.parseTime=Tr(this.props.ordering_keys_time_format),this.formatTime=Cr(this.props.ordering_keys_time_format),this.xtickFormat=function(e){return"object"==Za(e)?this.formatTime(e):n(e)}):(this.parseTime=null,this.formatTime=null,this.xtickFormat=n),this.xscale=De(),this.xaxis=dn().scale(this.xscale).tickSizeInner(4).tickSizeOuter(0).tickFormat((function(t){return e.xtickFormat(t)})).tickPadding(-18),this.ytickFormat=n,this.yscale=De(),this.yaxis=pn(an,undefined).scale(this.yscale).tickSizeInner(4).tickSizeOuter(0).tickFormat((function(t){return e.ytickFormat(e.invLinkFunction(t))})).tickPadding(2),this.xlabel.node().onchange=function(){return e.internalDraw()},this.ylabel.node().onchange=function(){return e.internalDraw()},this.svg.on("mousemove",(function(t){return e.mouseMoved(t)})),this.svg.on("click",(function(){return alert("This original index of the sample you clicked is "+e.nearestExpIndex)})),this.svg.on("mouseout",(function(t){return e.mouseOut(t)})),window.addEventListener("resize",this.redraw),window.setTimeout(this.redraw,50)}},{key:"componentDidUpdate",value:function(){this.draw()}},{key:"mouseOut",value:function(){this.hoverLine.attr("display","none"),this.hoverx.attr("display","none"),this.hoverxOutline.attr("display","none"),this.hoverxTitle.attr("display","none"),this.hovery.attr("display","none"),this.hoveryOutline.attr("display","none"),this.hoverGroup1.attr("display","none"),this.hoverGroup2.attr("display","none")}},{key:"mouseMoved",value:function(e){var t,n,r=this;this.hoverLine.attr("display",""),this.hoverx.attr("display",""),this.hoverxOutline.attr("display",""),this.hoverxTitle.attr("display",""),this.hovery.attr("display",""),this.hoveryOutline.attr("display",""),this.hoverGroup1.attr("display",""),this.hoverGroup2.attr("display","");var a=function(e,t){if(e=function(e){let t;for(;t=e.sourceEvent;)e=t;return e}(e),void 0===t&&(t=e.currentTarget),t){var n=t.ownerSVGElement||t;if(n.createSVGPoint){var r=n.createSVGPoint();return r.x=e.clientX,r.y=e.clientY,[(r=r.matrixTransform(t.getScreenCTM().inverse())).x,r.y]}if(t.getBoundingClientRect){var a=t.getBoundingClientRect();return[e.clientX-a.left-t.clientLeft,e.clientY-a.top-t.clientTop]}}return[e.pageX,e.pageY]}(e,this.svg.node())[0];if(this.props.explanations){for(t=0;t
WARNING:shap:Using 120 background data samples could cause slower run times. Consider using shap.sample(data, K) or shap.kmeans(data, K) to summarize the background as K samples. 0%| | 0/30 [00:00<?, ?it/s]
Have you run `initjs()` in this notebook? If this notebook was from another
user you must also trust this notebook (File -> Trust notebook). If you are viewing
this notebook on github the Javascript has been stripped for security. If you are using
JupyterLab this error is because a JupyterLab extension has not yet been written.
R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you’re looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don’t.
Continue reading: Gradient-Boosting anything (alert: high performance)
Analysis of Gradient Boosting Machine Learning
In this blog post, the author provides a detailed examination of decision trees in Gradient Boosting Machine Learning models. They compare how well AdaBoostClassifier works against their own implementation in the Cython-based Python package mlsauce.
The author then guides the reader through the installation and use of the mlsauce package, allows the reader to apply gradient boosting to any machine-learning model with fit and predict methods. It’s key to note that only regression algorithms are accepted as base learners and classification is regression-based.
Multiple models are used in the experiments, including xgboost, ExtraTreeRegressor, and RandomForestClassifier. Upon testing, the author concludes that there is still significant room for improvement, as all models are used with their default hyperparameters.
Long-term Implications and Future Developments
The long-term implications of this insight suggest a shift towards more adaptable machine learning models that are capable of handling different types of algorithms with improvement opportunities through hyperparameter adjustment.
In terms of future developments, the gradient boosting could be modified to handle other types of algorithms besides regression models, making it more readily adaptable to varying datasets and potentially increasing its utility in a broader range of applications.
Actionable Advice
If you’re working in the area of Machine Learning and you’re keen on achieving better performance, consider applying the gradient boosting technique to your models, especially if you’re using regression algorithms. You can observe the improvements for yourself by using the mlsauce package for your Python projects as demonstrated in the post.
One definite area for experimentation would be to tweak the hyperparameters of your models. Since they were left at default in the examples provided above, there is potential for improved performance if the parameters are optimised for your specific use case.
Lastly, keep an eye out for future updates to these techniques which might make gradient boosting suitable for other types of algorithms and not just regression models.