by jsendak | Jan 30, 2024 | DS Articles
LLMs aren’t the right tool for most business applications. Find out why — and learn which AI techniques are a better match.
Understanding the Scope of AI in Business Applications
The growing advancements in Artificial Intelligence (AI) have made it an increasingly compelling choice for many business applications. However, industry experts assert that LLMs (Language Model-based systems) might not be the right tool for most applications. This prompts the need to understand which AI techniques are a better fit and their long-term implications.
Limitations of LLMs
While LLMs offer impressive capabilities, they come with their share of limitations when applied in a business context. For one, they might not provide the desired level of accuracy in data analysis, especially when handling complex datasets. Furthermore, their often high maintenance costs and need for extensive training data make them a challenging solution for many enterprises.
The Better Matches: Other Techniques for Business Applications
Given the limitations associated with LLMs, other AI technologies present more suitable solutions for most business applications. This leads us to explore these alternatives and their potential benefits.
- Supervised Learning: This training of an AI model involves labeled input and output data, ensuring increased accuracy.
- Unsupervised Learning: Unlike supervised learning, unsupervised learning doesn’t rely on labeled data, making it useful for exploratory analyses.
- Reinforcement Learning: Here, the AI model learns how to make decisions based on the concept of rewards or penalties, ideal for real-time decision-making.
Long-Term Implications & Future Developments
Choosing the right AI tools and techniques could lead to significant long-term benefits, including improved performance, cost-efficiency, and scalability. Supervised learning might ensure more accurate predictions while unsupervised learning could help identify hidden patterns in data. Reinforcement learning could potentially enable better real-time decision-making.
As AI continues to evolve, it’s anticipated that more robust and efficient techniques will emerge, further enhancing the potential for business applications. This could range from advancements in deep learning to the proliferation of AI in day-to-day business operations.
Actionable Advice
Avoid relying solely on LLMs for your business applications. Instead, explore other AI techniques to find the best fit for your specific needs. Stay current and adaptable with AI advancements and regularly reassess your strategies.
Depending on the complexity of your data and your specific use case, different AI techniques may work best. Therefore, it’s important to understand each tool’s strengths and weaknesses to make an informed decision. Invest in training and talent development to ensure your team can effectively leverage these sophisticated tools.
Read the original article
by jsendak | Jan 30, 2024 | DS Articles
Natural language processing (NLP) is changing the manner in which we converse with one another as well as machines.
Long-term Implications and Future Developments of Natural Language Processing
Natural Language Processing (NLP) is a revolutionary technology that is reshaping the way humans and machines converse with each other, ushering in a new era of technological advancement. This domain of artificial intelligence is poised to bring about some unprecedented changes in both communication and data mining algorithms.
Long-term Implications of NLP
The long-term implications of NLP are vivid and far-reaching. As the NLP technology seeps into numerous industries, including customer service, finance, healthcare, and more, it can greatly enhance productivity and efficiency. Here are the key implications:
- Automation of Tasks: With NLP, routine tasks such as scheduling meetings, booking reservations or ordering food may be entirely automated, thereby transforming the way businesses function.
- Improved Customer Service: Bots equipped with NLP can answer queries round the clock, decreasing the waiting time for customers.
- Advanced Healthcare: NLP can facilitate analysing patient records and diagnosis more accurately, potentially impacting healthcare by saving countless lives.
- Data Mining: NLP can help in sorting through infinite data and derive meaningful insights from them easily, thereby significantly improving data mining techniques.
Future Developments in NLP
Given the rapid advancements in this field, the future of Natural Language Processing looks highly promising. There are several innovations that NLP might see in the coming years:
- Human-like Interaction: The future of NLP may see machines having conversations almost like humans. It could make interaction with machines more intuitive and less mechanical.
- Machine Translation: Machine translation might be enhanced through NLP to translate not just words but also intonations, cultural nuances, and non-verbal cues in various languages.
- AI Assistants: NLP may power AI assistants like Siri, Alexa or Google Assistant to understand more complex commands and respond with improved accuracy.
Actionable Advice Based on these Insights
Understanding the implications of NLP and its future prospects, businesses can act strategically to stay ahead in the competition. Following are some actionable insights:
- Invest in Technology: Businesses should consider investing in NLP technology not only to streamline operations but also to serve customers better.
- On-board Expertise: Companies must rope in experts who understand NLP so as to navigate feasibly through the increasing complexities of this domain.
- Focus on Training: For optimal use of the technology, businesses need to train staff to work alongside machines possessing NLP.
In conclusion, the world is moving closer towards achieving advanced machine-human conversational abilities, largely catalyzed by Natural Language Processing. This transformative change is set to redefine our communication methods and interaction with machines.
Read the original article
by jsendak | Jan 29, 2024 | DS Articles
[This article was first published on
shikokuchuo{net}, and kindly contributed to
R-bloggers]. (You can report issue about the content on this page
here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don’t.
A surprise
I came to write this post because I was surprised by its
findings.
The seed for it came from a somewhat obscure source: the Tokyo.R
slack channel.
This is actually a fairly vibrant R community. In any case, there was
a post by an R user suprised to find parallel computation much slower
than the sequential alternative – even though he had thousands, tens of
thousands of ‘embarassingly parallel’ iterations to compute.
He demonstrated this with a simple benchmarking exercise, which
showed a variety of parallel map functions from various packages (which
shall remain nameless), each slower than the last, and (much) slower
than the non-parallel versions.
The replies to this post could be anticipated, and mostly aimed to
impart some of the received wisdom: namely that you need computations to
be sufficiently complex to benefit from parallel processing, due to the
extra overhead from sending and coordinating information to and from
workers. For simple functions, it is just not worth the effort.
And this is indeed the ‘received wisdom’…
and I thought about it… and the benchmarking results continued to
look really embarassing.
The implicit answer was just not particularly satisfying:
‘sometimes it works, you just have to judge when’.
And it didn’t really answer the original poster either – for he just
attemped to expose the problem by using a simple example, not that his
real usage was as simple.
The parallel methods just didn’t work. Or rather didn’t ‘just
work’TM.
And this is what sparked off The Investigation.
The Investigation
It didn’t seem right that there should be such a high bar before
parallel computations become beneficial in R.
My starting point would be mirai, somewhat naturally (as
I’m the author). I also knew that mirai would be fast, as
it was designed to be minimalist.
mirai? That’s みらい or Japanese for ‘future’. All you
need to know for now is that it’s a package that can create its own
special type of parallel clusters.
I had not done such a benchmarking exercise before as performance
itself was not its raison d’être. More than anything else, it
was built as a reliable scheduler for distributed computing. It is the
engine that powers crew, the high
performance computing element of targets, where it
is used in industrial-scale reproducible pipelines.
And this is what I found:
Applying
the statistical function rpois() over 10,000
iterations:
library(parallel)
library(mirai)
base <- parallel::makeCluster(4)
mirai <- mirai::make_cluster(4)
x <- 1:10000
res <- microbenchmark::microbenchmark(
parLapply(base, x, rpois, n = 1),
lapply(x, rpois, n = 1),
parLapply(mirai, x, rpois, n = 1)
)
ggplot2::autoplot(res) + ggplot2::theme_minimal()
Using the ‘mirai’ cluster resulted in faster results than the simple
non-parallel lapply(), which was then in turn much faster
than the base default parallel cluster.
Faster!
I’m only showing the comparison with base R functions. They’re often
the most performant after all. The other packages that had featured in
the original benchmarking suffer from an even greater overhead than that
of base R, so there’s little point showing them above.
Let’s confirm with an even simpler function…
Applying
the base function sum() over 10,000 iterations:
res <- microbenchmark::microbenchmark(
parLapply(base, x, sum),
lapply(x, sum),
parLapply(mirai, x, sum)
)
ggplot2::autoplot(res) + ggplot2::theme_minimal()
mirai holds its own! Not much faster than sequential,
but not slower either.
But what if the data being transmitted back and forth is larger,
would that make a difference? Well, let’s change up the original
rpois() example, but instead of iterating over lamba, have
it return increasingly large vectors instead.
Applying
the statistical function rpois() to generate random vectors
around length 10,000:
x <- 9900:10100
res <- microbenchmark::microbenchmark(
parLapplyLB(base, x, rpois, lambda = 1),
lapply(x, rpois, lambda = 1),
parLapplyLB(mirai, x, rpois, lambda = 1)
)
ggplot2::autoplot(res) + ggplot2::theme_minimal()
The advantage is maintained!
So ultimately, what does this all mean?
Well, quite significantly, that virtually any place you have
‘embarassingly parallel’ code where you would use lapply()
or purrr::map(), you can now confidently replace with a
parallel parLapply() using a ‘mirai’ cluster.
The answer is no longer ‘sometimes it works, you
just have to judge when’, but:
‘yes, it works!’.
What is this Magic
mirai uses the latest NNG (Nanomsg Next Generation)
technology, a lightweight messaging library and concurrency framework – which
means that the communications layer is so fast that this no longer
creates a bottleneck.
The package leverages new connection types such as IPC (inter-process
communications), that are not available to base R. As part of R Project
Sprint 2023, R Core invited participants to provide alternative
commnunications backends for the parallel package, and
‘mirai’ clusters were born as a result.
A ‘mirai’ cluster is simply another type of ‘parallel’ cluster, and
are persistent background processes utilising cores on your own machine,
or on other machines across the network (HPCs or even the cloud).
I’ll leave it here for this post. You’re welcome to give
mirai a try, it’s available on CRAN and at https://github.com/shikokuchuo/mirai.
Continue reading: mirai Parallel Clusters
Understanding Parallel Computing and Future Developments in R
This post is a follow up on an informative article from R-bloggers which illustrated that parallel computing is sometimes mistakenly assumed to be slower than sequential computing. The article presented some insightful benchmark tests and threw light upon how a tool called mirai performs better. The key argument presented in the article is the case with simple or ’embarrassingly parallel’ code, which are functions with iterations or loops that can be executed simultaneously or independently. It points out that parallel computations in such cases can be speedier than non-parallel ones, contrary to what many believe.
Long-term Implications
The value of understanding when and how to leverage parallel computing is enormous. As computational workload spikes, whether due to more complex analysis demands or increases in dataset size, the use of parallel processing will become more and more common.
Especially, the part regular users of the mainstream R system, such as researchers, data scientists, and business analysts, will find being equipped with such knowledge and tools of tremendous value. They can significantly cut down the time it takes to process large datasets and complex calculations by properly leveraging parallel computing resources and tools like mirai.
The development and acceptance of mirai-like packages will also incentivise similar intuitive, user-friendly software packages that democratise access to high-performance computing.
Potential Future Developments
- Mirai could prompt updates in the default R system to optimise for parallel computations as it has evidently showcased its speed in benchmark tests.
- Mirai’s rise to broader usage could inspire the development of more specific packages leveraging parallelism for specific computations.
- These learnings may further feed into the development of intelligent systems capable of deciding when best to leverage parallel computation, optimising efficiency, and user experience.
Actionable Advice
Users who often compute ’embarrassingly parallel’ tasks should try and harness the power of mirai.
- Start by installing and using the package via CRAN or GitHub for parallel processing in lieu of lapply() or purr::map().
- Keep an eye on the evolution of tools like mirai and adapt your workflows accordingly. As software continues to evolve, so must our best practices.
- Take some time to understand when it is best to make use of parallel versus sequential computing. The article gives us some insights; a rule of thumb is to look at the complexity and size of computation job at hand.
Notes
A benchmarking exercise in the original article showed that the mirai package resulted in faster computations than both the default parallel cluster in R and non-parallel computations for execution of a simple function over 10,000 iterations.
The mirai package benefits from NNG (Nanomsg Next Generation) technology, a high-performance messaging library and concurrency framework. It applies new connection types such as IPC (inter-process communications), not available to base R.
Read the original article
by jsendak | Jan 29, 2024 | DS Articles
The post highlights real-world examples of NLP use cases across industries. It also covers NLP’s objectives, challenges, and latest research developments.
NLP: Future Developments and Long-Term Implications
Natural Language Processing (NLP) is revolutionizing various industries in an unprecedented manner. From healthcare to advertising, its practical functionalities are helping institutions make sense of human language, aiding in decision-making processes. This article evaluates the long-term implications of NLP and how it is positioned to redefine vertical markets in the coming years.
NLP Objectives and Challenges
NLP has a simple objective: making computer systems understand, interpret, and generate human language. However, accomplishing this feat involves overcoming numerous, intricate challenges like context recognition, colloquial interpretation, tone or sentiment discernment, and more. Also, each advancement must cope with the continuous evolution of human language itself.
The Future of NLP: Developments on The Horizon
Research is fervently underway in various realms of NLP, driving it towards a future where its implementation might be far more widespread and nuanced than currently envisioned.
- Semantic Understanding: Efforts are being made to improve NLP’s semantic understanding capabilities, which will allow it to contextually interpret sentences rather than just words.
- Sentiment Analysis: NLP aims to refine its sentiment analysis capabilities, which interpret the emotional tone of human communication. Better sentiment analysis could reshape areas like customer service and marketing.
- Multi-language and Dialect Understanding: Enhancing understanding and interpretation of multiple languages and dialects is a significant focus area of research. This development will make NLP more inclusive and globally applicable.
Long-Term Implications of NLP
The long-term ramifications of NLP are extensive, redefining the way businesses operate and communicate.
- Improved Customer Experience: By understanding customer intent and sentiment, businesses can provide personalized responses, thus improving customer experience exponentially.
- Data Analysis: NLP can help analyze vast volumes of unstructured data (like social media posts), providing valuable customer insights to companies, and forming strategic decisions.
- Global Expansion: With NLP’s ability to understand multiple languages, businesses can expand their services globally, erasing linguistic barriers.
Actionable Advice
Organizations of all types stand to gain significantly from integrating NLP into their operations. By anticipating NLP’s future developments, businesses should:
- Invest in AI and NLP technologies and training for both the immediate and long-term benefits.
- Consider NLP as part of their customer service strategy to improve the customer interaction experience.
- Appreciate the importance of data and leverage NLP for data analysis to drive improvements in their products and services.
In conclusion, NLP is increasingly becoming a pivotal part of business processes. The future developments on the horizon promise to deliver even greater possibilities, shaping a future where human-computer understanding is significantly nuanced and meaningful. Businesses that adapt to this transformative wave will be the ones setting the pace in their respective fields.
Read the original article
by jsendak | Jan 29, 2024 | DS Articles
Discover how Gen AI is transforming the modern financial landscape and get the list of the top 7 use cases of Gen AI in the FinTech industry.
Gen AI: Revolutionizing the Fintech Industry
The advent of Generation AI (Gen AI) is poised to revolutionize various sectors, with the Fintech industry being no exception. The transformative power of Gen AI lies in its ability to automate operations, reduce risk, custom-tailor user experiences, and forecast trends. This article will delve into the long-term implications of Gen AI in Fintech and the possible future developments we might witness.
The Potential Future Developments in Gen AI
While Gen AI is already making significant strides in Fintech, future advancements promise even greater efficiency and sophistication. Here are some of the predicted developments:
- Data Security: As reliance on technology grows, so does the threat of cybercrime. Increased sophistication in AI can lead to stronger anti-fraud measures and cybersecurity solutions.
- Precision in Trend Projection: With the continual enhancement of AI’s predictive capabilities, financial institutions can look forward to more precise trend projections and consequently smarter investment strategies.
- Personalization of Financial Services: Powered by AI, Financial service providers will be able to offer more personalized services tailored to individual customers’ financial goals and capabilities.
Long-term Implications of Gen AI
“The transformative power of Gen AI lies in its ability to automate operations, reduce risk, custom-tailor user experiences, and forecast trends.”
The long-term impact of Gen AI in Fintech is significant. It is expected to make financial services more efficient, personalized, and safe. This encourages wider accessibility and acceptance of Fintech solutions among common people.
Actionable Advice
Given the dominance and potential of Gen AI, the following actions are recommended:
- Invest in AI: Any business in the financial sector should consider investing in AI technology. It not only improves efficiency but also acts as a competitive differentiator.
- Prioritize Security: As technology advances, so do security threats. Fintech companies should prioritize cybersecurity to protect both business and customer data.
- Embrace Personalization: Understanding customers’ unique needs can foster loyalty and drive growth. Implementation of AI can help with this by delivering tailored financial services.
In conclusion, the continued adoption and advancement of Gen AI presents a transformative pathway for the world of Fintech. While the tech upheaval may seem overwhelming, embracing the change and riding on the wave of innovation promise a future of efficient, secure, and personalized financial services.
Read the original article
by jsendak | Jan 28, 2024 | DS Articles
[This article was first published on
R on Publishable Stuff, and kindly contributed to
R-bloggers]. (You can report issue about the content on this page
here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don’t.
In statistics, there are a number of classic datasets that pop up in examples, tutorials, etc. There’s
the infamous iris dataset (just type iris in your nearest R prompt),
the Palmer penguins (the modern iris replacement),
the titanic dataset(s) (I hope you’re not a guy in 3rd class!), etc. While looking for a dataset to illustrate a simple hierarchical model I stumbled upon another one: The cake dataset in
the lme4 package which is described as containing “data on the breakage angle of chocolate cakes made with three different recipes and baked at six different temperatures [as] presented in Cook (1938)”. For me, this raised a lot of questions: Why measure the breakage angle of chocolate cakes? Why was this data collected? And what were the recipes?
I assumed the answers to my questions would be found in Cook (1938) but, after a fair bit of flustered searching, I realized that this scholarly work, despite its obvious relevance to society, was nowhere to be found online. However, I managed to track down that there existed a hard copy at Iowa State University, accessible only to faculty staff.
The tl;dr: After receiving help from several kind people at Iowa State University, I received a scanned version of Frances E. Cook’s Master’s thesis, the source of the cake dataset. Here it is:
Cook, Frances E. (1938). Chocolate cake: I. Optimum baking temperature. (Master’s thesis, Iowa State College).
It contains it all, the background, the details, and the cake recipes! Here’s some more details on the cake dataset, how I got help finding its source, and, finally, the cake recipes.
The cake dataset
The cake dataset can be found in
the lme4 package with the following description:
Data on the breakage angle of chocolate cakes made with three different recipes and baked at six different temperatures. This is a split-plot design with the recipes being whole-units and the different temperatures being applied to sub-units (within replicates). The experimental notes suggest that the replicate numbering represents temporal ordering.
So for each of the $3 times 6 = 18$ recipe and temperature combinations, Cook made 15 (!) replicates, resulting in a total of $3 times 6 times 15 = 270$ cakes/datapoints. Here’s the first couple of rows:
| replicate |
recipe |
angle |
temperature |
| 1 |
A |
42 |
175 |
| 1 |
A |
46 |
185 |
| 1 |
A |
47 |
195 |
| 1 |
A |
39 |
205 |
| 1 |
A |
53 |
215 |
| 1 |
A |
42 |
225 |
| 1 |
B |
39 |
175 |
| … |
… |
… |
… |
If you want the full dataset without getting lme4 here’s the cake dataset as a CSV file. Plotting this dataset we can quickly conclude that the cake breakage angle increases as a function of baking temperature:
While the cake dataset is found in lme4, the original source is Cochran and Cox’s book Experimental designs. But what’s the original original source? Any why measure the cake breakage angle?
The hunt for the source of the cake dataset
From the lme4 documentation I knew that the cake dataset came from the study by Cook (1938) but no amount of Googling, Binging, nor Google Scholaring resulted in any trace of a digital copy.
I did find that physical copies existed at Iowa State University and at Cornell, which presented a problem for me, being physically in Sweden.
There was an option to request that the copy would be digitized, an option available to Iowa State faculty only.
Twitter to the rescue, I thought, and fired away a tweet that got a tumbleweed response.
But, final proof for me that Twitter is dying, the same request on Mastodon (
come join me!) was an astounding success!
I got many helpful responses, with several pointing me directly at Iowa State staff that might help me out. Like this one from
Karl Broman:
A quick e-mail later and I got this very encouraging e-mail from Dan Nettleton at the Department of Statistics, Iowa State:
He recruited the help of Philip M. Dixon, Department of Statistics, and Megan O’Donnell, Research Data Services Lead, and after a couple of days more I got this from Megan:
She (the busy Research Data Services Lead with a looming deadline) is apologizing to me (the random Swede with an eccentric cake thesis digitization request) that it took a few days to get me everything I asked for!? Still, the feeling of shame for having wasted Megan’s time was overshadowed by joy. Attached to the e-mail was, of course, also the full Master’s thesis of Frances E. Cook from 1938: Chocolate cake: I. Optimum baking temperature..
Highlights from Chocolate cake: I. Optimum baking temperature
Reading the thesis, it’s immediately clear that the breakage angle of cakes wasn’t the main focus. Instead, Cook was after some “accurate scientific information” on the optimum baking temperature for chocolate cake.
To figure out what was the best chocolate cake, she needed a battery of measures of cake goodness, such as cake tenderness, as measured objectively by its breaking angle. There were also several subjective measures, as found in the “Score Card for Cake” on page 50.
But how was the breaking angle of the cakes measured? In the thesis, we learn that “The tenderness of the cake was tested with the breaking angle apparatus as described by Myers (1936)”, but there are no images that show us how it functioned. While I can’t find an online trace of Myers (1936) I do believe I’ve found a description of this very apparatus in Lowe and Nelson (1939)!
From an outsider perspective, not being active in the field of culinary research myself, the thesis of Cook comes off as being fantastically serious about cake. I especially adore that it includes photographs of all the cakes:
But, to be fair, in the photos above, you can clearly see how the baking temperature influences the volume of the cake.
The cake recipes
Like in a food blog that has been SEOed to death, here, finally, at the very end, are the cake recipes. I might not be the most experienced cake maker, but this is by far the most complicated chocolate cake recipe I’ve ever seen.
Now, for the baking time and temperature above you get a matrix of options.
The answer for which option to pick can be found a bit further down in table XV, which displays the total scores for each option.
The winner, when considering the dimensions texture, tenderness, velvetiness and eating quality, was Recipe C with a baking temperature of 225 C° (437 F°) for 24 minutes. I’m no cake scientist, but if a linear model is to be believed when extrapolating outside of the range of the dataset (always a good idea) this cake would be delicious when baked in a pizza oven!
Continue reading: The source of the cake dataset
Uncovering The Significance of The Chocolate Cake Dataset
The Chocolate Cake Dataset, available in the lme4 analytics package for studying a hierarchical model, reflects how the breakage angle varies according to the baking temperature and recipe used. Despite its convenient presence in this library, it appears there’s more to this set of data than initially meets the eye. Deeply steeped in historical roots, this dataset comes from a study conducted by Frances E. Cook in 1938.
The Origin of the Cake Dataset
The foundation of this dataset is a master’s thesis titled “Chocolate cake: I. Optimum baking temperature” written by Frances E. Cook at Iowa State University. After hunting down this surprisingly rare digitized resource, it was discovered that the dataset portrays different baking scenarios that Cook explored to uncover the ideal baking temperature for chocolate cake.
Dataset Composition and Methodology
In the study, Cook created cakes using three different recipes and baked them at six different temperatures. She then evaluated various properties of the resulting cakes, such as their breakage angle and subjective measures like tenderness and velvetiness. The dataset is organized with 15 replicates for each of the 18 recipe-temperature combinations, leading to 270 total data points!
Predictions and Conclusions
An analysis of this data presents an interesting finding – the cake breakage angle increments as the baking temperature rises. Besides, Recipe C baked at a temperature of 225°C for 24 minutes was rated highest for texture, tenderness, velvetiness and overall eating quality, making it an ideal choice for chocolate cake lovers!
Long-term Implications
The cake dataset has implications beyond its original context, extending its value to the fields of statistical modeling and machine learning. It could serve as a ‘real-world’ example while learning hierarchical modeling, regression analysis, and other statistical algorithms due to its intuitive structure and comprehensible variables.
The Future
Looking forward, it’s conceivable that the classic datasets like the cake dataset will continue to be a key feature in the data analysis, statistics, and machine learning fields. These age-old datasets serve as important tools in teaching valuable data skills and can be used in developing new analytics software.
Actionable Advice
For academicians, students, or organizations keen on using such datasets or studying them further, here is some advice:
- Reach out to academic institutions: As shown in this exploration of the cake dataset’s origins, many valuable resources are tucked away in the libraries of academic institutions. Don’t hesitate to reach out for search assistance.
- Open-source databases: Check open-source platforms for old datasets. Platforms like R provide a plethora of datasets for different usages.
- Learn from these datasets: Use these practical and easily understandable datasets for learning and teaching concepts reliably.
Read the original article
︎
