AddMaple Blog

There’s a lot we could talk about but I’ll try to keep it brief. We’re closely tracking OpenAI’s advancements and currently use OpenAI in AddMaple to summarize data containing free text (unstructured data) among other things. We have a few additional features coming soon and are convinced that AI coupled with programming genius can fundamentally improve data analysis, making it more efficient, intuitive, and even enjoyable. But why stop there? Why not innovate beyond data analysis and improve how we write and read, or engage with reports too? More on that below. In short, OpenAI’s first DevDay developer conference didn’t unveil GPT-5, but it brought super exciting opportunities.

Longer context support
This is something we’ve grappled with at AddMaple. If you want a LLM to summarize a lot of free text, then the best way is to send it everything in one chunk. However GPT3.5 and 4 maxed out at 32,000 tokens (about 24,000 words). This is still a lot, but wasn’t enough for larger surveys or datasets containing long free-text data, e.g. interviews. While there are existing ways of splitting the text up and creating multiple summaries and then summarizing the summaries, this was slow and introduced other problems.

The new GPT4 Turbo (vroom 🏎 )now supports about 96,000 words or 128,000 tokens. This is more than a typical novel which contains about 80,000 words. While pretty expensive, it unlocks many more use cases (and kills Anthropic’s main USP…)

GPTs and the Assistants API with knowledge retrieval and persistent threads
A large number of AI startups were effectively threatened yesterday with this feature. Without getting too much into the weeds of the tech, OpenAI has created their first version of “AI Agents”. These agents are slightly different to the ChatGPT interface where you ask it a question and it generates a response. Agents can essentially perform more advanced tasks.

Here are a few ideas of what can be created with this feature (generated by ChatGPT of course…)

A Custom Family Cooking Assistant: rather than stop with ChatGPT’s ability to generate recipe ideas, you could have a more advanced cooking assistant that is tailored to your family’s dietary preferences, what you have in the fridge and even the kitchen gadgets that you own/enjoy using.

An Intelligent DJ That Gets Your Music Taste: We use Spotify A LOT at home, especially their auto-generated playlists. But there are 2 huge problems. Firstly, it sometimes suggests styles of music that I really don’t like, and I can’t easily stop Spotify from playing those. And secondly, Spotify sometimes loses track of, tracks I’ve heard on different playlists within a short period. I also don’t really want to hear a song more than 2 or 3 times in a week regardless of playlist. It should be fairly easy to create an assistant that can fix these issues but can still help me discover more music.

A Legal Precedent Researcher: Were you to be in possession of a hefty database containing previous legal cases, you could benefit from a Legal Assistant that can help you find relevant cases containing similar precedents to the one you’re working on currently.

From a business perspective, OpenAI is really interesting. They are going after enormous markets simultaneously and by all accounts are succeeding in all of them!

D2C: ChatGPT which is already boasting nearly $1 billion annual revenue

Custom Models: which allows enterprises and well-funded startups to train their own models with pricing starting between $2–3 million: https://openai.com/form/custom-models

API Platform: OpenAI keep reducing their API pricing, enticing developers from big and small companies alike to build on top of their services. Some of the new features they’ve offered start bringing an element of “lock-in” which will make it even harder for their competitors.

Revenue-sharing and Incentivization: OpenAI announced its launch of a marketplace for AI assistants within ChatGPT, offering developers a slice of the revenues they generate.

I’m interested in what you took away from the announcements yesterday?

For data analysis and AddMaple specifically these are some of the ideas we are exploring:

An Insights Assistant to support users with their data analysis:

Our current capabilities— AddMaple detects column types and converts raw data into chart summaries automatically without requiring upfront cleaning/preparation. Users can near-instantly segment/pivot and share these charts individually or add them to a report for their audience to explore.

But with the updates to OpenAI, an Insights Assistant could interpret survey results from AddMaple’s visualizations and could provide natural language summaries of key findings to speed up report writing. For instance, once AddMaple visualizes customer satisfaction across different regions or products, the assistant could automatically generate a text summary detailing which regions/products received the highest and lowest satisfaction.

Once AddMaple reads the dataset provided and provides in-browser summaries, we could allow users to could ask a Query Assistant specific questions about their survey data, such as, “Which features received the highest satisfaction ratings?”. The assistant could process this query, retrieve the relevant visualizations from AddMaple, and could then further assist the user by providing additional verbal or written summaries of the charts. Our published reports are currently explorable too. With OpenAI’s announcement, our reports could be queried using natural language assistants too. Not everyone reading a report wants to read. What if they could query the report and the published columns using natural language? Sounds like a great opportunity to me.

There’s a lot we could talk about but I’ll try to keep it brief. We’re closely tracking OpenAI’s advancements and currently use OpenAI in AddMaple to summarize data containing free text (unstructured data) among other things.

OpenAI’s first developer conference happened. Here’s what we thought of DevDay

We'll look at what Text Analysis is, the challenges with it and how it is made simpler with generative AI

In today's data obsessed culture, significant resources are invested in the collection, security, and storage of data. From meticulously gathering customer feedback to ensuring the highest standards of data protection, organizations spare no cost in managing their data assets. 

However, amidst this focus on data management, the analysis of free text—a gold mine of qualitative insights found in support tickets, sales calls, product reviews, and survey responses—frequently remains an afterthought. In 2023 the IDC (the International Data Corporation) published a white paper which found that 90% of the data generated by organizations was unstructured, and only 10% was structured. The vast majority of data processing and analysis budget goes towards structured data. According to IDC research, only half of an organization’s unstructured data is analyzed to extract value. If all data is to be considered as an asset — as echoed by many executives — then it must be treated as such. 

This oversight means that a wealth of valuable insights remains untapped, hidden within the vast volumes of unstructured text that businesses accumulate over time. Organizations which learn to use automation and AI to synthesis these vats of insights will almost certain gain the advantage. 

Text analysis is augmented with Gen AI in AddMaple to bring clarity to complex unstructured data. 

As cornerstones of qualitative research, Thematic and Categorical analysis provides structured methods for developing, analyzing and interpreting patterns across a qualitative dataset which involves systematic processes of coding/tagging to develop themes or categories. 

AddMaple offers two approaches to text analysis using Gen AI

Thematic Analysis uses codes to tag text data into specific units of meaning. Each code is distinct and differentiated, allowing consistent tagging of relevant text. These codes are then grouped to identify overarching themes and patterns within the text. These themes often reveal deeper meanings beyond the surface. 

Categorical Analysis uses codes to organize text data into clearly defined classes according to type. The codes are also distinct and differentiated to allow the text to be classified into buckets. Categorical analysis is more surface level - e.g. group this text by verbs, adjectives, subject mentioned, sentiment expressed. Categorical Analysis deals with the semantics of what was said.  


Let's use Lego bricks to illustrate the difference between these approaches. 

Imagine you have a basket of unsorted Lego bricks representing unstructured text data. The image below shows how you would sort, organize and analyze the Lego bricks using both approaches. 
 
Left: The Lego bricks are arranged in a work of art for artistic expression to convey a message representing Thematic Analysis which reveals meaning in text data. 
Right: The lego bricks are arranged into compartments by color and type representing Categorical Analysis. 

If we use a Thematic approach to sorting our Lego bricks, we would group these bricks into patterns or shapes in a quest for meaning until we are eventually able to interpret these into broader themes to tell a story or stories.

If we use a Categorical approach to sorting our Lego bricks approach, we organize them by type. We might even organize them multiple ways by different criteria, e.g. by color, by size, by year of release and combinations of those. AddMaple supports multi-way coding for multidimensional analysis of text data. This means you can code a text column for sentiment, subject/topic, feature requests etc. Each time you code data in AddMaple, a new column is created with your codes that you can use to segment the data. This is something we couldn't do with our Lego bricks because we would only be able to put one brick in one compartment. But digital data can be tagged/coded multiple ways. 

This brings us to why analyzing unstructured data manually is difficult. We usually only code data in one dimension if at all. Most organizations report not analyzing unstructured data at all. Let's dive into the common reasons and how Gen AI can help us overcome these. 

Challenge 1: Language Barriers in Global Data

One of the foremost challenges in analyzing qualitative data as members of a global village is the diversity of languages your data might occur in, including a wide array of alphabet characters and scripts which many tools don't support. 

Traditional text analysis methods often require separate coding staff and processes for each language and script, leading to increased complexity and the potential for inconsistency. This linguistic maze can significantly hinder the analysis of global surveys for example. When free text coding spans across people, tools and processes, it becomes challenging to maintain a coherent and unified analysis approach. 

AddMaple solves this with the support of LLMs which are able to understand multiple languages at once and analyze such a rich body of multi language and multi script text data at once, using the same code book. This means the codebook contains universal codes with descriptions that LLMs use to consistently categorize text data across different languages, ensuring consistency. For example, a code for "customer complaints" would apply to relevant comments written in English, Spanish, Chinese, etc based on the same criteria defined in the codebook.

Challenge 2: Most qualitative analysis tools don't facilitate segmentation

Let's look at free text responses in a survey as an example of missed segmentation opportunities. Oftentimes, this text data isn't analyzed at all but merely visualized in a word cloud. But let's say the verbatim data is coded and grouped into themes, these themes aren't usually brought back into the survey dataset to slice or segment the survey with for deeper learning. The reason for this is typically because free text responses are analyzed in a different tool than the data analysis tool used for the survey itself. Qualitative researchers might code the text data in a qualitative tool or in Excel using colors, but it is rare that the codes derived from the qual analysis are brought back into the survey dataset to segment or triangulate the survey data itself.

This siloed approach of qual analysis as a separate exercise from quant analysis restricts the depth of insights that can be gained, as it overlooks the potential correlations between textual responses and other data points within the survey. Not only can free text responses reveal insight about something we might not have known to ask in the first place, the insights uncovered as a result of the free text analysis might correlate with something else in the data, which would be overlooked unless the codes are used to segment or cross tabulate the data. In AddMaple you can code large amounts of free text responses in minutes, and then use our stats engine automatically reveal how the codes relate to other columns. You might realize that certain themes have a significant relationship with respondents job titles, something that would have been lost in traditional text analysis tools and approaches. 

To illustrate what we mean, we have used the published dataset from UX Tools 2023 Design Tools survey and analyzed the final free text question, "Any final comments or feedback about the survey?" 

We use Gen AI to read the text and suggest suitable codes. 

We use Gen AI again to assign the codes row-by-row to each comment. 

Once the data is coded, we use AddMaple's stats engine to automatically reveal how columns are related. 

We see that Product Designers were significantly less likely to answer words to the effect of 'No Comment' to this question, likely indicating that they are the most invested in this survey compared to people working in other roles. 

Challenge 3: Text is Usually Coded With a Single-Focus Lens

Usually thematic or categorical coding of text looks at one focal point with which to code the data. Let's look at the same example of free text analysis we used in point 2 previously. The question read, "Please leave any feedback about the survey or final comments". The standard approach would be to code the responses according to feedback of the survey and the topics covered in the general comments. This is above board. But what if we could use another coding lens too?

For example the same question could be re-coded to look at sentiment with buckets ranging from very negative to very positive. This could then be used to cross tabulate the data with another column, e.g. let's look at the final comments by their role at work or their organization size. 

Below we look at one-lens with which we can code the open-ended survey question: "Any final comments or feedback about this survey?" 

However, this example question is fairly broad as are the verbatim responses. We could also code it for emotion analysis, intent analysis, sentiment analysis, semantic analysis, depending on what we want to learn. By looking at these comments in a word cloud, we see the various approaches we could take. 

AddMaple's word cloud is interactive, you can zoom in and out by filtering by one or two, three words etc. The word cloud shows us a variety of directions we could take. We could analyze this data according to descriptive patterns by looking at the adjectives used, such as interesting, helpful, long, fun, amazing, excited, love, feel, great appear. We also see industry related words such as tools, portfolio, process, appear which could lead us to code by UX design tasks. We also see specific UX Tools mentioned by brand such as Figma, Sketch, Userbrain, Protopie and more, which means we could also analyze by brands mentioned. We see industry related words mentioned such as designer, job, business, prototype - so we could look at this in context of the UX industry. We also see words that could express user intent such as move, add, coming, question, find, option. 

Word Cloud 1: Shows all meaningful words from the answers to the question. We are looking at single unit words, or unigrams. There were 680 responses to this question. 

Word Cloud 2: Shows all meaningful words for comments containing the word 'survey'. There were 86 comments containing the word 'survey'. We see all related words for those specific comments. 

Word Cloud 3: Shows all meaningful words for comments containing the words 'survey' AND 'great'. There were 14 comments containing those two words, or bigrams. We see all related words from those 14 comments too. 

Read more about AddMaple's text mining via interactive word clouds that give you adaptive cruise control in the form of unigrams, bigrams, trigrams here.

Due to the time and effort it takes to code manually, a rigid coding structure is often preferred or specified in order to obtain repeatable and comparable results. This can make sense, where qualitative studies are repeated periodically. 

However, this fixed approach (deductive) can be preferred simply due to the complex nature of coding unstructured data and often does not accommodate the dynamic and iterative process of developing a coding schema that truly reflects the nuance and complexity of the data at hand. Many surveys that started decades ago have codebooks that don't fully address the new issues of today as an example. 

Researchers might be expected to fit their insights into pre-defined codes simply due to external constraints, rather than by what is best for the thematic or categorical analysis itself, which can stifle the discovery of new themes and categories and can limit the adaptability of the analysis to evolving research insights. 

A tool that is flexible to work within an existing framework AND flexible to support the researcher who wishes to uncover new codes/meaning would offer the best of both worlds. With AddMaple, one can code using the existing codebook AND code again using a supplemented codebook with newly added codes. Because we use Gen AI, this won't be a drain on time and researchers could learn twice or thrice as much in less time than when coding manually. 

Challenge 5: Collaboration Barriers in Resource-Limited Settings 

Best practices in thematic analysis and categorical analysis often recommend a collaborative approach, where multiple researchers review codes, themes, and concepts to ensure reliability, clarity and to reduce the risk of limited perspectives. But it is not always possible to work within a research pair or team. This scarcity of resources can lead to challenges in maintaining the quality and depth of analysis, as the collaborative review process is crucial for validating and enriching the thematic framework. In AddMaple, we augment the researcher by having the AI suggest codes, the human review/edit and add codes and we have the AI apply the codes iteratively together with the human researcher. In this way, we give researchers an AI research assistant, and with it, we are able to introduce additional perspectives/opinions at scale thanks to LLMs. 

Side note: We refer to researchers' values, opinions, and experiences as perspectives, rather than simply labeling them as bias. Read more about this here

Challenge 6: Thematic Analysis Tools Don't Always Count the Codes

In some instances counting the amount of times certain codes appear, isn't helpful, particularly when conducting thematic analysis. But in many cases a code count can be useful. 

When is it not helpful to count codes? 
Some research projects search for meaning, regardless of the amount of times this idea or view was expressed. One interviewee might express an idea that is profound and sheds light to the topic being studied in such a way that it impacts the results profoundly even though those ideas were in the minority across research participants. 

Here, all instances of distinct meaning are significant merely by virtue of being expressed. And this will be conveyed in the report, regardless of instances or count frequency. In qualitative analysis we recognize that numbers in of themselves, don't necessarily mean something is more true, more significant or more worthy of our attention. 

When is it helpful to count codes?
It is often helpful to quantify codes in order to gauge the urgency, prevalence or importance of items/tags/codes within text data. For example, in reviewing exit surveys and open ended answers where people type in their reasons for unsubscribing to a product, having a count of the number of times certain reasons were given, is useful. Without counting the codes, the impact of the analysis will be limited because it will be difficult to know which factors had the greatest bearing on a customer cancelling for a given period. This is why we simply count the amount of times codes occur within the qual dataset in AddMaple, but you can ignore these counts of course! 

And as mentioned above, AddMaple also applies statistical significance testing to the text analysis codes to show our users when those codes relate to other columns. For example, let's say we're analyzing exit surveys for a webapp and one reason people gave for leaving was 'the app is slow'. AddMaple will automatically highlight how those codes relate to other columns and were there to be a connection with say, browser type, we will reveal that for you. 

AddMaple's Text Analysis Approach - deeply human with Gen AI baked in 

And this brings us to AddMaple and specifically how we help you get structure from unstructured data.

By directly addressing the challenges above, AddMaple's text analysis tool not only offers solutions to the inherent limitations of traditional methods but also enhances the thematic and categorical analysis process, making it more accessible, flexible, efficient, and insightful for researchers dealing with complex, multi-lingual, and multi-dimensional data sets. We created our tool in collaboration with researchers and factored in the constraints they find themselves working within. 

While AddMaple uses generative AI, this doesn't mean the AI will code the data a different way each time you open your project, nor does it mean that the researcher is left out of the coding process all together. 

We've designed a dance of sorts, between researcher and AI assistant. A workflow that adapts according to the task at hand, the data being analyzed and is enjoyable, repeatable, accurate and intentional to let the researcher lead the process throughout! 

AddMaple uses the latest advancements in Generative AI and with it, the ability to apply real world knowledge to help researchers organize unstructured text. 

Below is a screenshot of unstructured data from AfroBarometer's survey. AfroBarometer asked respondents living in Africa to write down the names of the countries with favorable economic models. Respondents listed countries in their own language. Gen AI with its contextual world knowledge understands that french speaking respondents from Cameroon and Senegal etc, use the name Allemagne for Germany. This would be time consuming for a researcher and would require them to have some knowledge of the African languages used by survey respondents. We put AddMaple's ability to categorize unstructured text to the test and asked it to sort the countries listed in the answers into continents. 

Does AddMaple supports Open (Inductive) and Closed (Deductive) Coding? 

Yes! When we developed this tool, we ensured that both Open (Inductive) and Closed (Deductive) coding approaches can work! Because both approaches are useful for different purposes. If we've lost you, skip to the sections below for details on what these approaches are, why each brings value and when to apply them. 

The Inductive Approach: The data leads the exploration without preconceived codebooks or frameworks

The inductive approach analyzes qualitative data without preconceived ideas, codes, notions or theoretical frameworks. It allows researchers to immerse themselves in the data, allowing patterns, themes, and categories to emerge due to the researcher's engagement with the data, iteratively. Big Q TA goes beyond summarization of what was said, to meaning - this is more than summarization. As in, is not a passive approach where codes and themes jump out at the researcher but instead through reflexivity and time, deeper themes emerge that would not have been obvious. 

This bottom-up approach is particularly useful in exploratory research where the aim is to uncover new insights or to study a new or less understood topic. It can capture complexities, contradictions, nuance and produces rich data. The disadvantage is that it is time consuming and requires the researcher to keep revisiting generated codes periodically. It also requires that codes be revisited whereby codes that are too similar be combined, and where duplicate codes within other themes be extracted and the themes themselves be reconsidered. 

Use case for inductive thematic analysis: 
It could be used to study how people experience and use a brand new technology. Existing codes and frameworks won't exist yet. Without imposing any predefined categories, researchers can allow participants' own experiences to guide the identification of codes, and the patterns and themes that emerge. As new ideas come up, it may prompt the need for further data collection whereby the researchers have to conduct further interviews until they are satisfied that they understand enough. At the heart of the coding, theme development and interpretation of meaning, will be the active researcher, reflecting, considering and remaining open to understanding meaning within meaning. Without restraining results within a construct. 

In conclusion, text analysis represents a critical, yet often underutilized, component of data management. The challenge of processing unstructured text—such as customer feedback, survey responses, and other free-text forms—is significant due to its volume and complexity. AddMaple addresses these challenges by employing generative AI to enhance both thematic and categorical text analysis. This approach facilitates deeper insights by enabling efficient coding and analysis across multiple dimensions and languages simultaneously, without the need for separate tools or coding teams for different languages.

AddMaple simplifies the text analysis process by allowing users to automatically code text data, integrate these codes within larger datasets, and perform multidimensional analyses quickly. This reduces the time and effort traditionally required for manual coding and allows for more dynamic and responsive analysis to emerging data trends.

By integrating AI, AddMaple not only streamlines the workflow but also ensures consistency and depth in the analysis of qualitative data. We encourage researchers and organizations dealing with large volumes of unstructured data to consider AddMaple as a tool to unlock the valuable insights lying dormant within their data, making the process more accessible and insightful.

We'll look at Categorical Analysis and Thematic Analysis of free text, the challenges with it and how it is made simpler with generative AI

Analyzing Free Text Is Much Easier with Generative AI

Comma-Separated Values (CSV) files are common in data management and analysis, providing a simple format for storing tabular data. If you have access to spreadsheet software such as Excel or Google Sheets,  opening smaller CSV files might be straightforward. However larger datasets or CSV files with non-standard settings present a challenge. 


In this article we will explore how you can open large CSV files online with AddMaple. 

Before we dive into the solutions, let's take a moment for a technical aside. CSV files, despite the name, don't strictly require commas as separators. Depending on the regional settings and specific requirements, semicolons, tabs, or other characters can serve as delimiters. Moreover, the way CSV files handle line breaks varies across different operating systems—Windows uses a carriage return and line feed (CRLF), while macOS and Linux typically use just a line feed (LF). These nuances can mean that your CSV file looks wrong when loaded into software not designed to handle this wide range of options.


Another common issue arises with quoted content. CSV standards allow for fields to be enclosed in quotes, enabling the inclusion of line breaks, separators, or other special characters within a field. However, inconsistent implementations can lead to discrepancies in how this content is handled, potentially resulting in data misinterpretation or errors when opening files.

AddMaple offers a fast, easy to use solution for opening and analyzing CSV files directly in your browser. There is no need to download and install any software. AddMaple also provides the following advantages of cloud based spreadsheet services:

Handling Large Files: AddMaple is designed to efficiently open large CSV files, up to 1GB, without overloading your system. This capability is particularly beneficial for users who have experienced the frustration of spreadsheet applications freezing or crashing under the weight of hefty datasets.

Privacy and Security: With AddMaple, your data remains securely on your device. Unlike some online tools that require you to upload your files to a server, AddMaple processes data locally, ensuring that your sensitive information is not exposed to cloud vulnerabilities.

Versatility and Accessibility: AddMaple's browser-based nature means you can access your data from any device with internet connectivity. Whether you're in the office or on the go, your CSV files are just a few clicks away.

To open a CSV file online using AddMaple, simply register for a free account. You can use your Google account or create an account with your email address.

Once your account is created, create a new project with your CSV file.

AddMaple will efficiently load your CSV file (regardless of the settings it was created with), detect your column types and create explorable summaries.

Benefits of Opening CSV files in AddMaple

When working with CSV files that have a large number of rows or columns, efficiently managing and analyzing the data can become quite challenging. AddMaple addresses these challenges by offering a set of features designed to simplify the data analysis process.

One of the key functionalities of AddMaple is its ability to automatically generate summaries for each column in a CSV file. This feature is particularly useful for gaining an immediate understanding of large datasets, as it provides a high-level overview of the data contained within each column. Whether dealing with quantitative or qualitative data, these summaries can help identify trends, outliers, or patterns at a glance.

In addition to summarizing data, AddMaple creates explorable pivot tables for each column. This allows users to interact with the data in a more dynamic way, facilitating deeper analysis without the need to manually set up pivot tables. Users can easily pivot their data based on different variables to explore relationships and dependencies between columns, which is especially helpful when navigating datasets with numerous variables.

We’ve recently added a suite of AI powered tools to AddMaple. Simply load your CSV and you can:

Get AI powered explanations of charts. AddMaple can give you insightful commentary on your charts.

Get AI powered summaries of text data. This is invaluable for drawing key insights from large volumes of free text data.

Use AI to generate and apply codes (tags) for thematic analysis. Turn your qualitative data into quantitive data with the power of AI. This feature can also be used for sentiment analysis, for example turning reviews into likert scale data.

Try opening your CSV file with AddMaple right now, if you have any issues don't hesitate to let our support team know.

Learn how to open CSV files online with AddMaple, including support for large files and non-standard encodings.

How to Open CSV Files Online

In an age where technology evolves faster than a reality TV star's career, it's baffling — and frankly a bit embarrassing — to see spreadsheets still lumbering through our offices like digital dinosaurs. Yes, spreadsheets. Those sprawling, soul-sucking grids that have clung to the world of data analysis with the tenacity of a reality show contestant refusing to accept they've been voted off. They’re the typewriters of the digital era: quaint, nostalgic, and utterly out of their depth in the face of modern demands. As much as we love a bit of retro charm, let’s face it: spreadsheets may be the jack of all trades, but they clearly are the master of none.

Financial Analysis and Budgeting: Spreadsheets can be error-prone and lack audit trails, making them risky for complex financial tasks. They also struggle with real-time data integration and advanced predictive modeling, which are crucial in dynamic financial environments.

Data Storage and Organization: As a data storage solution, spreadsheets lack robustness. They don't offer the relational database capabilities, data integrity checks, and security features provided by database management systems, leading to potential data corruption and security risks.

Reporting and Data Presentation: While spreadsheets can create basic charts and graphs, they fall short in providing advanced visualization and interactive reporting capabilities found in specialized data visualization tools, limiting the depth and interactivity of data presentation.

Statistical Analysis and Calculations: For statistical analysis, spreadsheets offer limited functionality. They can't handle large datasets efficiently and lack advanced statistical features and accuracy compared to dedicated statistical software, leading to potentially flawed analyses.

Project Planning and Management: Spreadsheets are not ideal for dynamic project management as they lack features like real-time collaboration, automatic update capabilities, and integration with other project management tools. This can lead to outdated information and inefficient management processes.

Spreadsheets hold a special place in the pantheon of computing history, standing as one of the first graphical software programs to gain widespread adoption in the business world. Introduced in the late 1970s with VisiCalc, followed by Lotus 1-2-3 and eventually Microsoft Excel, these spreadsheet programs were revolutionary. They transformed the computer from a tool for specialists into an indispensable aid for a broad range of business professionals.

By presenting data in a visually intuitive grid format and allowing users to manipulate and visualize numbers graphically, spreadsheets bridged the gap between the abstract world of programming and the concrete needs of business planning and analysis. This innovation laid the groundwork for the graphical user interfaces that we take for granted today, marking a significant leap in making computing accessible and practical for everyday users.

Spreadsheets, hailed as the jack-of-all-trades in the data world, have long been the default tool for various data-related tasks. From financial modeling to basic data analysis, their ubiquity is undeniable. However, take a step back and the shortcomings of spreadsheets, especially for certain data analysis tasks, become increasingly apparent. This article explores why spreadsheets, despite their popularity, may not be the best tool for many modern data analysis needs.

One of the primary appeals of spreadsheets is their perceived simplicity. They offer a familiar grid layout, easy data entry, and basic calculation capabilities. However, this simplicity can be deceptive. For complex data analysis tasks, spreadsheets can become unwieldy and inefficient. They often require extensive manual input and manipulation, which not only is time-consuming but also increases the risk of errors.

Spreadsheets are fundamentally limited in their ability to process large datasets. As data volume grows, spreadsheets become slower, less responsive, and more prone to crashing. This limitation severely hampers their utility in an age where data is growing exponentially in both size and complexity, making them unsuitable for tasks like big data analysis or real-time data processing.

In today's interconnected world, the ability to collaborate effectively on data projects is crucial. Spreadsheets, however, fall short in this aspect. While there have been improvements with cloud-based options, they still lack the robust collaborative features found in more specialized data analysis tools, making teamwork and data sharing more cumbersome.

Spreadsheets can inadvertently lead to the creation of data silos. Each spreadsheet becomes an isolated data repository, disconnected from the broader data ecosystem of an organization. This fragmentation hinders comprehensive data analysis, as data from different spreadsheets often needs to be manually consolidated, a process prone to errors and inconsistencies.

A fundamental design aspect of spreadsheets that often goes unquestioned is their default editability. Every cell in a spreadsheet is readily editable, which, while providing flexibility, poses significant risks, especially in data analysis contexts. The ability for any user to alter data, intentionally or accidentally, can lead to data integrity issues. In scenarios where data should be sacrosanct, the editable nature of spreadsheets is more of a liability than an asset.

The ease of editing in spreadsheets makes it difficult to trace the history of changes, leading to potential data integrity challenges. In a data analysis process, understanding the lineage of data — where it originated, how it has been transformed, and who has modified it — is crucial. Spreadsheets, with their editable nature, lack robust mechanisms to track these changes, making it challenging to ensure the accuracy and reliability of the data analysis.

Contrast this with more specialized data analysis tools, which often treat data as read-only by default. These tools typically separate data input from data analysis, ensuring that the raw data remains unaltered. Any transformations or analysis are performed on copies or views of the data, preserving the original dataset's integrity. This approach is fundamental to reliable data analysis, as it safeguards against unauthorized or accidental alterations. 

Pivoting - Powerful but needlessly complex

Pivoting in spreadsheets is a classic example of a feature that has been retrofitted into an existing interface, rather than being intuitively designed from the ground up. While pivot tables are powerful tools for summarizing and analyzing data, their integration into the spreadsheet environment is far from seamless. Users often find themselves navigating through a maze of floating tables, jumping between different sheets, and dealing with a plethora of settings and options that can be overwhelming even for experienced users.

The user experience of creating pivot tables in spreadsheets highlights several design inefficiencies. First, the process of setting up a pivot table is not intuitive, often requiring multiple steps and adjustments to display the desired data correctly. This setup becomes even more cumbersome when dealing with large datasets or complex pivot operations. Additionally, the resulting pivot tables and charts are typically 'floating' entities within the spreadsheet, leading to a cluttered and disorganized workspace. Switching between different sheets to cross-reference data adds another layer of complexity, disrupting the workflow and increasing the chances of errors. This disjointed experience stands in stark contrast to what a well-designed, user-centric data analysis tool should offer.

While pivot tables in spreadsheets provide valuable data analysis capabilities, their implementation and the user experience they offer are far from ideal. They exemplify how adding features to a tool not originally designed for complex data analysis can lead to a convoluted and inefficient user interface. As data analysis becomes more central to organizational decision-making, the need for tools specifically designed for these tasks, with user experience as a core consideration, becomes increasingly evident.

It's time to reassess our longstanding attachment to spreadsheets. While they have been groundbreaking in their time, paving the way for the democratization of data analysis, the technological landscape has shifted dramatically. A good user experience is no longer optional, it is required. Today's data challenges require more than just a grid of cells and basic formulas; they demand a sophisticated, visual-first approach that can only be provided by specialized tools designed specifically for modern data analysis.

Tools like AddMaple make data analysis fast (and fun) with intuitive interfaces, powerful visualizations, and the ability to handle large, complex datasets with ease. They are not just about presenting data; they are about unlocking stories hidden within numbers, revealing insights that spreadsheets can only skim the surface of. 

In embracing these specialized tools, we're not just upgrading our software; we're upgrading our mindset. We're acknowledging that the future of data analysis lies in tools that are as dynamic and multifaceted as the data itself. Let's move forward, acknowledging the limitations of spreadsheets, and step into a future where data analysis is visual, intuitive, and enjoyable.

In a world awash with bespoke digital solutions, clinging to spreadsheets for complex tasks is akin to using a typewriter in the age of laptops – quaint, nostalgic, and woefully inadequate. They are the digital equivalent of a Swiss Army knife used for open-heart surgery.

The Terrible UX of Spreadsheets

A Deep Dive into Ngrams, Filters, and Contextual Analysis

Before you roll your eyes at the mention of word clouds, please let us explain why ours are useful with the addition of a few extra tricks. We can all agree that analyzing unstructured text is difficult. And we can agree that free text data could contain hidden insights. For example that open ended survey question might have brought something to your attention you didn't know to ask about in the multiple choice questions. The review data waiting to be explored might help you understand how your customers view your product compared to your competitors. And getting a handle on support data could help you prioritize the next feature because you have a greater understanding of that customer need in the first place

Exploring unstructured free text (e.g. free text responses to a survey question) is difficult. At AddMaple we’ve explored a few different approaches and would like to share why we think interactive word clouds are an extremely powerful tool.

There is a lot of evidence that “conversational surveys” or allowing users to enter free text answers is valuable. Rather than restricting users to choosing a preselected list of answers (which by definition will be limited to what we know at the time), free-text answers allow you to discover what users really think about a topic. As well as surveys, analyzing customer support messages or customer reviews is extremely valuable. Oftentimes a decision is made to leave out open questions because of the analytical challenges they bring but dear reader, we have made this much simpler to tackle. 

At AddMaple we are big believers in thematic analysis (and the role that AI can play in it), however, in this post, we will explain a simpler and quicker way to explore a large free text dataset. We’ll explore n-grams, stemming, and interactive word clouds and we’ll explain why we made the choices that we did.

N-grams are commonly used in natural language processing, here is a brief explanation of what they are:

N-grams: A contiguous sequence of 'n' items from a given text sample. In text, an item often means a word. The 'n' in n-gram denotes how many words are grouped together, aiding in understanding language structure.

Unigrams: Single words. In the phrase "I love ice cream", unigrams are "I", "love", "ice", and "cream".

Bigrams: Pairs of words. From the same phrase, the bigrams are: "I love", "love ice", and "ice cream".

Trigrams: Sequences of three words. Using the aforementioned example, trigrams are: "I love ice" and "love ice cream".

By using n-grams, we can capture linguistic patterns, with unigrams highlighting individual words, while bigrams and trigrams offer contextual insight by examining words in pairs or groups of three.

N-grams, Unigrams, Bigrams, and Trigrams Explained

Traditionally, many analysts have championed the use of bigrams and trigrams when trying to understand the thematic structure of a text. These combinations of two or three words can provide context that a single word - or unigram - might lack. For instance, the phrase "climate change" encapsulates a specific concept, while the words "climate" and "change" separately might not capture the full essence.

But here lies a challenge: What if crucial words that are conceptually linked don't sit right next to each other in a sentence? Take, for instance, a statement like "Climate is experiencing drastic changes due to global activities." The words "climate" and "changes" are separated, and thus wouldn’t be included together in bigrams or trigrams.

Combining trigrams, bigrams, and unigrams using a scoring approach that ranks trigrams the highest, followed by bigrams and finally unigrams. The purpose of this technique is to highlight common bigrams and trigrams as well as unigrams. In the above example, this would have resulted in a bigram of “Stack Overflow”. This approach has some issues, however:
undefinedundefinedundefined

Allowing the user to select between unigrams, bigrams, and trigrams. This gives the user control, however, it still has issues:
undefinedundefined

After exploring the above (and other approaches), we settled on using unigrams but combined with interactive filters.

Word clouds that use unigrams and interactive filters are the most useful in our experience. Here's how it works: when you click on a word, the cloud can adjust to show you other words most frequently associated with your chosen term. This interactivity means that, regardless of whether the words are immediately adjacent to each other or several words apart in the original text, you can gain insights into their contextual relationships.

Imagine being able to tap on "climate" and immediately seeing words like "change," "impact," and "global" becoming more prominent, thus giving you a clear snapshot of the surrounding discourse. With the flexibility to filter down on two or more words, users enjoy the benefits that bigrams or trigrams traditionally offer, but with the added advantage of broader context and real-time exploration.

AddMaple is optimized to apply filters within milliseconds for datasets of up to a million records. The user experience improvement of instant filters is remarkable. It allows quick analysis and exploration of datasets in a way that is just not possible when each filter takes a few seconds to apply. 

 After settling on unigrams, we still had the issue of stemming to consider. 

Why we choose not to use stemming in our interactive word clouds

Stemming has long been a cornerstone in the realm of textual analysis. At its core, stemming aims to reduce a word to its base or root form, allowing words like "answer" and "answers" to be processed as one entity. By consolidating such variations, analysts can gain a clearer picture of the frequency and prominence of core themes within a body of text.

However, like many tools, stemming is not without its complications. For the uninitiated, a word cloud that displays only stems might seem alien and incoherent. Let's take the word "computing" for example. Its stem, "comput", would baffle many, as it doesn't correspond to any commonly used term in English. While "answer" and "answers" are aggregated as "answer", not all words are so accommodating.

To avoid the confusion of only including the stem, we tried displaying the most frequent variant of a stemmed word in the word cloud. On the surface, this seemed like a viable solution. Yet it introduces a few issues. Firstly there are many situations where you do want to see variants of a stem, for example you may not want the noun “researcher” and the verb “researching” to be linked together. In addition it creates these issues when filtering:

Filtering solely on the most frequent variant misses the other variants of the stem

Filtering on the stem alone catches unrelated words, e.g. the stem "operat" could pull in "operation", "operator", and "operate", but could also unintentionally capture words like "operative".

Cleverly filtering on all variants of a stemmed word works but makes text filtering more complex both from a filtering perspective and for the UI. It breaks the simple connection between the text filter box and the text filtering.

In light of these challenges, our approach at AddMaple has been to side-step stemming in favor of harnessing the power of interactive filtering in word clouds. When users can swiftly sift through terms and observe relationships in real-time, the distinction between "answer" and "answers" isn’t a problem. The intuitive nature of this method, coupled with its speed, makes it a good choice for exploring and understanding free text data.

In addition to our decision not to use stemming in our interactive word clouds, we also chose to forgo lemmatization. Lemmatization, like stemming, aims to reduce words to a base or dictionary form. However, it considers the context and converts the word to its meaningful base form, unlike stemming, which often chops off word endings. For instance, "better" is lemmatized to "good" and "running" to "run". While this sounds helpful, especially for linguistic accuracy, it introduces complexities in a word cloud context. The transformed words can sometimes deviate significantly from their original form, potentially confusing users who are trying to connect the visualized words back to their raw data. 

Additionally, the computational demands of lemmatization, especially for large datasets, can impact the responsiveness and simplicity of the user interface, a critical aspect of our design ethos at AddMaple. Thus, while acknowledging the linguistic sophistication of lemmatization, we chose to prioritize user experience and immediacy of insight, steering clear of any processes that could complicate or obfuscate the intuitive understanding of data.

After exploring more complex natural language processing, and observing how these options impacted on the ease with which a user is able to grasp the meaning of a body of text, and the user-interface itself, we settled on keeping things simple. 

The design of user interfaces is as much an art as it is a science. While advanced functionalities are appealing technically, they can also obfuscate what the user is trying to do, muddying functionality, leading to unintended confusion. It's a delicate balance between equipping users with powerful tools and ensuring clarity and intuitiveness. Overcomplicate the interface, and you risk alienating users; oversimplify, and you may not deliver on the promise of deep insights.

This is why at AddMaple, we have centered our approach on the principle of intuitive interaction. We’ve designed our word clouds to look good, be intuitive to work with and be quick to dive into the details and explore connections between words.

We also don’t hide the actual raw free text responses from users - once a dataset has been filtered down using our interactive word clouds it is easy to switch to a table view focussed on the column being analyzed.

In our word clouds, we exclude "stop words," which are common but less informative words like 'the', 'is', and 'on'. This helps highlight more meaningful words, making the data analysis clearer and more relevant. Currently we only support English stop words, but let us know and we can add support for your language.

Load the Data: Begin by loading your dataset into AddMaple. You can load from CSV or SAV files or bring your data in directly from Typeform, Survey Monkey or Google Drive. AddMaple will automatically detect your column types and prepare text columns for unigram based analysis.

Instant Word Cloud Visualization: For each text column, a word cloud is auto-generated. Crafted from unigrams, this cloud disregards stop words and captures the essence of the responses by highlighting the most frequently used terms. Whether respondents are discussing generic concepts or mentioning specific product names, you'll be able to see the key words used at a glance.

Interactive Exploration: Dive deeper into the data by clicking around. Employ the filtering options to narrow down topics or themes. The beauty of AddMaple's interface is the fluidity it offers; see a word you are interested in, click it and instantly see related words. 

Deep Dive with Table View: After honing in on a subset of data through filters, explore the raw data in our table view. By simply clicking on the 'table' link, you're presented with a neatly organized table showcasing the raw responses related to your filters. Each filtered word is highlighted to make it easy to see how they were actually used in the raw data. This focused approach, devoid of other distracting columns, makes it easy to explore lots of raw text responses in an efficient manner.

AddMaple's intuitive design takes you on a journey from overarching themes to the nuanced sentiments of individual respondents. This gradient of insights ensures that while you grasp the bigger picture, you never lose touch with the unique perspectives that often hold the key to understanding your audience better.

If you think we should approach this problem another way, we’d love to hear from you. Designing a data analysis tool that is powerful, but intuitive is a big challenge. Hopefully this post explains the why behind our word cloud interface. The next time you have a dataset with free text responses try exploring it with AddMaple.

Explore how interactive word clouds can be a useful tool for analyzing unstructured text data. In this article we disccuss the technical choices we made in building this feature from ngrams to stop words, stemming and lemmatization.

The Power of Interactive Word Clouds

We're all used to streamed text content from LLM chat APIs. With the release of OpenAI's text-to-speech API we wanted to explore how to stream audio at the same time as text.

A lot of chat APIs use server-sent events. This works well for text based data and there are nice primitives to use in the browser to handle these event streams.

However we wanted to stream text and binary audio data at the same time. One option could have been to use base64 encoding, but this felt hacky and inefficient. It involves extra processing on both the server and the client.

With the help of GPT4 we discovered the multipart/x-mixed-replace content type. This used to be natively supported in browsers for streaming images, e.g. from webcams. Most browsers no longer support it directly for image/video data but you can still use fetch and handle the stream manually.

This is perfect for our use case as it means we can send different content types in a single stream. For GPT4 this can include text, images, audio and JSON.

This article is a walk through of implementing this in a NextJS app - full code here.

This is a quick demo of the code in action, with full duplex audio and multiple simultaneous tool (function) calls.

Demo open ai.mp4

multipart/x-mixed-replace what does it look like?

It is actually simple. You define a boundary string and then separate each part of the stream with that boundary. Each part has a content type header followed by the data (in whatever format you want).

We wanted to start playing audio as soon as text was being streamed back from the chat completions API (rather than wait for the full response). A quick implementation might send each chunk of text to the TTS API. However this would be inefficient and result in very unnnatural audio as often OpenAI will send a single word at a time. So we wrote a simple function that sends a minumum of 30 characters and only full sentences to the TTS API.

Each returned audio stream is then pushed to the client. The end result is a mixed stream that contains both text and audio data. Audio is sent a sentence at a time, while text is streamed a word at a time with new partitions every 5 words. The result is a stream that could look like this:

----MultipartStreamBoundary
Content-Type: text/plain

How can I help you
----MultipartStreamBoundary
Content-Type: text/plain

today?
----MultipartStreamBoundary
Content-Type: audio/mpeg

���� BINARY DATA ����

A longer response would have the text and audio interspersed and contain 100s of text and audio sections.

Sending the boundary string and content-type header after every 5 words of text isn't efficient. Seeing as we are in control of both the client and the server we could probably do something cleverer, however for a demo this works well enough.

Each time the client receives a chunk of data it checks for the encoded boundary string. When it finds a boundary it processes the data buffered so far. It first checks for a content-type header, if it is text data it updates the messages array with the content, if its audio data it queues it up to play and if its JSON data it runs the associated functions.

Here is the (GPT4 generated and lightly edited) code to handle the stream, splitting it at each boundary string. Notice that because we're expecting binary and text data we keep the processing at the binary level rather than converting to text.

const encoder = new TextEncoder()
const boundary = encoder.encode(BOUNDARY)
const response = await fetch(url, {body, headers, method: "POST"})
const reader = response.body.getReader()
let {value, done} = await reader.read()
let buffer = value
if (done) {
  handlePart()
}

while (!done) {
  // Search for the boundary in the buffer
  let boundaryIndex = indexOfUint8Array(buffer, boundary)

  while (boundaryIndex === -1 && !done) {
    // We haven't found the boundary, read more data
    ;({value, done} = await reader.read())
    if (value) {
      buffer = concatUint8Arrays(buffer, value)
      boundaryIndex = indexOfUint8Array(buffer, boundary)
    }
  }

  if (boundaryIndex >= 0) {
    // Process the part we have found
    const part = buffer.slice(0, boundaryIndex)
    handlePart(part)
    // Remove the processed part from the buffer
    buffer = buffer.slice(boundaryIndex + boundary.length + 2) // +2 for the \r\n
  }
}

All the code for processing and handling this data is here.

The actual processing of each chunk of data then looks like this:

const {contentType, content} = getContent(part)
if (contentType === "text/plain; role=user") {
  // transcribed text from an audio recording
  addMessage({role: "user", content, id})
} else if (contentType === "text/plain") {
  addMessage({role: "assistant", content, id}) // handles concatening streamed text
} else if (contentType === "audio/mpeg") {
  const audioBlob = new Blob([content], {type: "audio/mpeg"})
  enqueueAudio(audioBlob)
} else if (contentType === "application/json") {
  const results = await Promise.all(content.map(handleFunction))
  addMessage([
    {role: "assistant", id, content: null, tool_calls: content},
    ...results.map((result, idx) => ({
      role: "tool",
      content: result,
      send: true,
      tool_call_id: content[idx].id,
    })),
  ])
}

In the above code we are dealing with 4 different types of content in just a few lines of code. Compare this to the code that handles receiving function/tool responses in vercel's ai library.

Remarkably playing the audio in the browser doesn't take many lines of code.

const audioBlob = new Blob([rawBytes], {type: "audio/mpeg"})
const audioUrl = URL.createObjectURL(audioBlob)
const audio = new Audio(audioUrl)
audio.play()

We ended up with a fairly simple hook that implements an audio queue where each audio chunk is played sequentially. The end result sounds like a single audio stream.

The newer OpenAI models support calling multiple functions (now called tools) in a single response. Now that we have a way of sending text and audio from the server in a single stream it made sense to also send JSON. There is no point streaming the individual parts of a function argument in the way that they are received from OpenAI. So we decided to buffer up any tool responses and send them as an application/json chunk. This makes the client implementation simpler as it now only receives complete JSON. Arguably its semantically better as well - our stream can now send 3 different types of data each with their own content type.

This demo shows off the power of multiple tools by defining separate functions to change the styling of a circle, square and triangle. It is a lot more powerful being able to ask the LLM to do multiple things at the same time.

Currently DALL-E isn't integrated into the chat completions API, but apparently thats coming soon. When it does there will be another content type we can add to our flexible multipart/x-mixed-replace stream 😊

After finding how easy playing audio is in modern browsers we decided to implement recording as well - complete with silence detection and waveform rendering.

The code needed for recording and sending to the server is super simple. Full code is here, but here is a simplified extract:

const stream = await navigator.mediaDevices.getUserMedia({audio: true})
const recorder = new MediaRecorder(stream, {mimeType: "audio/webm"})
recorder.ondataavailable = (event) => {
  const body = new FormData()
  body.append("file", event.data, "audio.webm")
  fetch(url, {body})
}

This was a fun experiment that we think could be useful for many projects. It seems highly likely that as a tech community we are going to be building a lot of software on top of LLM APIs. It also seems highly likely that more LLMs will become multi-modal AND that no matter the speed improvements there will be many use-cases where you want to stream the data to your application rather than wait for the full response.

Maybe its time for the return of multipart/x-mixed-replace!

ps we're regularly launching AI based features into smart data analysis platform - AddMaple

Exploring audio and text streaming from chat APIs, the use of the multipart/x-mixed-replace content type with multi-modal chat models enables efficient delivery of mixed content types like text, images, audio, and JSON in a single stream.

(ab)using multipart/x-mixed-replace for multi-modal chat streams

There is a treasure trove of interesting data available from the Pew Research Center

From the American Trends Panel Datasets to surveys exploring how we interact and relate to technology.

Some Pew Research data is available for free (not their most recent datasets) but the catch is that they are only available in the .sav (spss) format. This format is designed for IBM SPSS software, which is expensive and hard to use. But fret not. 

If you are keen to explore a SAV file, such as what you can get from Pew, but don't have IBM SPSS, then this guide is for you. We'll show you how to open, analyze, explore and create shareable charts and reports from Pew Research data in just a few clicks with AddMaple.

You might want to bolster your research portfolio, or use a report as a case study that requires you to have a reputable data source. Well, Pew Research Center is considered by many to be a trusted source for data measuring a broad range of issues, attitudes and trends shaping our world. It is a nonpartisan fact tank. Their research spans numerous topics, from politics and media to science and society. Their commitment to public service and open access means that many of their data sets are freely available for download. However, while the data is available, accessing its insights isn't always straightforward for the layperson. That's where AddMaple comes into play.

First things first, why should you use Pew Research data?

Learn how to use AddMaple to analyze datasets from Pew Research Center. No need for SPSS or other advanced statistics tools. AddMaple gives you an exploratory chart summary dashboard you can pivot, segment, share, add to a report and more.

How to explore Pew Research datasets that are free to download

Read the latest news and updates from the AddMaple team. We'll share our thoughts on data visualization, data analysis, and everything in between.