Python in Excel alternative: Open. Self-hosted. No limits.

My thoughts on Python in Excel

2024-06-11T00:00:00+00:00

UPDATE: xlwings Lite, my answer to Python in Excel, is now available in the official Excel add-in store for free.

On August 22, 2023, Microsoft revealed the preview of “Python in Excel”. As creator of xlwings and author of the O’Reilly book Python for Excel, I was obviously curious to give it a try. Since you shouldn’t judge a book by its cover, I decided to take a deep dive and record a LinkedIn Learning course on Python in Excel: Working with pandas DataFrames. Along the way I discovered many bugs and other issues and reported them on GitHub. All of them are either resolved or at least acknowledged/commented by now, which is fantastic, and I only wish it was as easy to report a bug in the Office.js and VBA world.

So what do I think about Python in Excel? Here’s the TL;DR:

We wanted an alternative to VBA, but got an alternative to the Excel formula language
Integrating the Jupyter notebook cells inside the Excel grid was a mistake
Python in Excel isn’t suitable for Python beginners nor for interactive data analysis
Right now, there are too many restrictions (can’t use your own packages and can’t connect to web APIs)
Here are the current use cases I see for Python in Excel:
- Computationally intensive things like Monte Carlo simulations
- AI stuff via the included packages (scikit-learn, nltk, statsmodels, imbalanced-learn, gensim)
- Advanced visualizations via Matplotlib/Seaborn
- Time-series analysis (this is one of Excel’s blind spots)
- Not sure about data cleaning/data analysis: since you almost certainly need Power Query, it may actually be simpler and faster to just stick to Power Query (instead of using Power Query and Python in Excel together)

Before we dive into the details, I want to clarify that this is just my personal opinion and is not meant to be a rant or critique. I’ve been in contact with the Excel team a few times along the way, and they are all super friendly and I value their interest in my opinion. Unlike me, they have to work with almost 40 years of Excel history and a ton of corporate overhead. Given these restrictions, they are doing an excellent job!

(Also check out the related discussions on HackerNews and Reddit.)

Make something people want
The mistake: placing the Jupyter notebooks cells inside the grid
Python in ~~Excel~~ the Cloud
A glue language without glue
Matplotlib is great, but a little interactivity wouldn’t hurt
No access to the Excel object model
DataFrames are great, but so are NumPy arrays and lists
A better alternative than “an alternative to the Excel formula language”
What are my plans with xlwings?
Conclusion

Make something people want

The Python in Excel story started in 2015 when user “Daniel” posted the following idea on Microsoft’s UserVoice page (which has been migrated to their new feedback forum in the meantime):

The idea quickly became the most popular one clearly wanting:

A scripting language
User-defined functions

Unfortunately, Python in Excel doesn’t support either of these two features. Instead of being an alternative to VBA, it’s designed to be an alternative to the Excel formula language. While I don’t think there’s anything wrong with Excel’s formula language, Python in Excel is really just a 2-dimensional Jupyter notebook inside the Excel grid. It uses a real Jupyter notebook kernel, by the way, that runs in the Azure cloud. This means that you can use notebook magics such as %%timeit, which is a cool thing!

John Lam, a Principal Architect on the project, told the Real Python Podcast that this implementation was just thought to be “a crazy idea”, but eventually it was picked by those people who call the shots at Microsoft. In my opinion, it should have remained a crazy idea, as I will explain in the next section.

The mistake: placing the Jupyter notebooks cells inside the grid

Python in Excel could have been straightforward by integrating a Jupyter notebook as a task pane in Excel. A Python developer would feel right at home, with the benefit that they can directly read and write from the Excel cells. On the other hand, Excel users can start dabbling around with Python in a “standard” environment and don’t have to learn a new tool when they move on from Excel. So what’s wrong with placing the notebook cells inside the grid?

It looks like a duck, but it ain’t one

Everyone knows how a spreadsheet works: you put in numbers and formulas in the cells, and everything just works because Excel figures out the dependencies of each cell and recalculates those that need to be updated when a specific input changes. It certainly doesn’t matter where on the spreadsheet these cells are.

Well, Python in Excel breaks this core spreadsheet truth: despite behaving very similarly to user-defined functions (“UDFs”), PY cells are evaluated from left to right and top to bottom. This includes the sheets, so the first sheet gets calculated before the second sheet, etc.

Do repeat yourself

When Office Scripts was introduced, it was such a relief that the code (unlike with VBA) was stored outside of the Excel workbook on SharePoint/OneDrive. All of a sudden, you could use the same code with multiple workbooks, without having to do any copy/pasting. This also means you can fix your code in exactly one location when you discover a bug. Life is good. Actually, it would have been even better if the Office Scripts file would be a regular TypeScript file that could be tracked on GitHub, but that’s a different story.

Now, Python in Excel stores the code in the Excel file itself. To the end user, the code lives in the cells, but technically, it’s stored under /xl/pythonScripts.xml inside the Excel workbook (i.e, in the ZIP file). So back to copy/pasting, yay!

Magic, magic everywhere!

Whenever you want to write out a pandas DataFrame (or any other Python object) to the Excel cells, you need to switch from Object to Values mode (the button on the left of the formula bar). There are 2 issues with that:

When the DataFrame uses the default integer index, it’s hidden when turning it into values. If that index is either text or non-default, it is shown. Magic!

Imagine that you have 100 PY cells, and in cell 3 you set the id column as DataFrame index. This means that in cell 100, where you convert the Python object into values, you all of a sudden have one column more.
Switching from Object to Values mode changes the formula from =PY("...", 1) to =PY("...", 0). This is, however, hidden from the user as the user only sees the Python code in the cell (the "..." part of the formula). Magic!

I see two main issues with that:
- It prevents you from referencing a cell with a Python object directly, as you’re always running the risk that someday, someone switches that cell to Values mode, which would break any formula that references this cell in object mode.
- PY cells in object mode behave differently than any other object cell (“Excel (rich) data types”). Take the stocks data type: the cell is always an object and you can show its price by doing =A1.Price in a different cell. PY cells could behave the same: if you want the values, you would write =A1.Values. A little consistency goes a long way.

Non-interactive data analysis

I primarily use Jupyter notebooks for interactive data analysis or for trying things out in incremental steps. You type a few lines of code in a cell and hit Shift+Enter to evaluate the cell, look at some intermediate result and continue with the next cell.

With Python in Excel, I don’t see the output of the cell, unless I:

Change the cell type from Object to Values. That’s often not possible as this requires having enough space around the cell.
Click on the preview icon, which opens a pop-up window. That’s a lot of work (even with the Ctrl+Shift+F5 keyboard shortcut) and the preview window often isn’t all that helpful: it’s too small and the values are shown in the Excel cell representation instead of how a Jupyter notebook would show it. This is supposed to be fixed at some point, but right now, almost 1 year after the initial version, it still shows #N/A in the preview so I don’t know if that is a np.nan, pd.NA, or pd.NaT.

Too much friction in the workflow means I am just not considering Python in Excel for interactive data analysis, unfortunately!

Diagnostics pa(i)ne

I would call the diagnostics pane the biggest offender in Python in Excel. It starts with the unnecessarily technical name (why not simply call it Output like in VS Code?).

Anyhow, the diagnostics pane shows whatever you print() in your cells as well as any error messages. So whatever a Jupyter notebook shows directly underneath the cell. But since Python in Excel pulls the output away from the cell, you have to manually associate the output back to the proper cell. Here are two examples:

When looking at a DataFrame, typically the first thing you run is df.info(). Now, since this prints the output, whenever Excel recalculates the Python cells, the Diagnostic pane shows up. That’s not just annoying, it’s also completely impossible to quickly relate the output on the diagnostics pane with the corresponding cell. It also keeps the output from the last run to make it even harder to find the lines that you’re after (a monospaced font would also help, by the way). So which output is my df1, and which one is my df2? I have no idea unless I look up the code that is sitting in cells L2 and K2:
The same happens with exceptions. Try to find the error in the following screenshot (the error happens in K2, which has already disappeared at the bottom of the diagnostics pane)

As a beginner Python coder, a crucial thing is to get easy feedback on your errors. That’s not the case here, which is why I don’t consider Python in Excel a good solution for Python beginners.

As an idea, the preview pop-up could be a much better place to show the output without loosing the context. Also, coloring the cell or the PY icon in red when there’s an error could help further.

Python in Excel the Cloud

As you’ve probably heard, Python runs in the cloud on Azure Container Instances and not inside Excel. There was a lot of backlash on social media about that (“privacy, privacy!”), but what these commenters are missing is that the majority of corporations already trust Microsoft with all their data by using OneDrive/SharePoint.

In fact, I agree that the cloud is the best way to bring Python to Excel. The maintenance and installation of a local Python environment simply isn’t possible for the inexperienced and when teams are bigger than a handful of people. The only other option is Web-Assembly (WASM), but it looks like the Excel team left this as an exercise for me to implement (see the end of this post).

What you need to keep in mind though is that Python in Excel will cost an extra subscription fee on top of Microsoft 365 (we don’t know how much yet) and there seems to be a quota: while I was recording my LinkedIn Learning course, I was greeted by this message:

While this is annoying enough when you’re supposed to record your videos, it’s pretty much unacceptable when you use your Excel spreadsheet for trading at a Hedge Fund (yes, that’s still a thing). And even if there would be no quota, the Azure cloud sometimes goes down and your Python cells won’t work—something to keep in mind.

All this to say that I wouldn’t use Python in Excel for something critical as long as I can’t switch to a self-hosted or local server when the cloud is down.

A glue language without glue

Much of the reason why people love Python is that it’s a “glue language.” With Python you can connect what needs to be connected, as there’s a package for everything. So a typical workflow might look like this: you download a few files from a web API, get data from a database, push everything through a pandas or polars DataFrame and finally send out an email with the results.

Well, such a basic workflow isn’t possible with Python in Excel. You can’t install any additional package and your code runs in a steel box, meaning that you can’t connect to the outside world via web API or database connection. The official solution is to use Power Query to acquire external data, as Python in Excel can then reference a Power Query. But really, one of the reasons I want Python and pandas in Excel is specifically to avoid Power Query (I don’t want to learn the M language), so I am not a big fan of this solution.

Here’s what I don’t understand:

Python in Excel runs on Azure Container instances. And Azure Container Instances can use your own container with all the dependencies you wish for, so I can’t really understand why the Excel team said in one of their Reddit AMAs that the dependencies would have to be installed every time from scratch when somebody opens the workbook. That’s just BS as the container image is built once and then sits there with all the dependencies ready when somebody opens the workbook next time.
If I can access web APIs via Office Scripts, why not with Python in Excel? As far as I know, the Office admin can restrict what Office Scripts can access, so I’d expect something similar to arrive with Python in Excel one day.

In summary, missing packages and blocked internet access means that you can’t use Python in Excel for many of the use cases that had people scream for Python in Excel in the first place.

Matplotlib is great, but a little interactivity wouldn’t hurt

Python in Excel offers Matplotlib and Seaborn (which is based on Matplotlib) to create plots where Excel charts may be too basic. That’s great and definitely needed, but the output is a boring static picture. xlwings has offered this for the last 10 years. What I am missing here is access to the modern interactive charts like Plotly, Bokeh, Altair, HoloViews, etc.

No access to the Excel object model

People use Excel for reporting all the time. And a super common use case is to consolidate one or more workbooks with data on multiple sheets. For example, there’s a sheet for each month or store location or department. With Python in Excel, you have to select the data on each sheet manually to get ahold of all the data:

jan = xl("Jan[#All]", headers=True)
feb = xl("Feb[#All]", headers=True)
mar = xl("Mar[#All]", headers=True)
...
df = pd.concat([jan, feb, mar, ...])

Good luck if you have weekly data or 100 departments to consolidate. If Python in Excel would give you access to the Excel object model (like VBA or Office Scripts), it would look something like this (the sample shows xlwings syntax):

data = []
for sheet in workbook.sheets:
    data.append(sheet.tables[0].range.options(pd.DataFrame).value)
df = pd.concat(data)

And talking about reporting: everyone wants their own color and cell border formatting, but without access to the Excel object model, that’s out of reach, too.

DataFrames are great, but so are NumPy arrays and lists

As soon as you select more than 1 cell, the xl() function will deliver a pandas DataFrame. I don’t mind this being the default, but it still happens on a very regular basis that you need a list, a NumPy array (or a polars DataFrame!) instead of a pandas DataFrame. And really, this should be easy to get. In fact, you can write your own converter today that solves this problem, only I don’t want to copy/paste this to every workbook…! Here’s how you’d fix this:

def myconverter(x, headers=False, convert=None, **kwargs):
    if convert is None or convert == pd.DataFrame:
        return excel.convert_to_dataframe(x, headers=headers, **kwargs)
    elif convert == np.array:
        return np.array(x)
    elif convert == list:
        return x
    else:
        raise ValueError(f"{convert} is not supported.")

excel.set_xl_array_conversion(myconverter)

Running this code in the Initialization pane (once it’ll be editable, for now place it in cell A1 of the 1st sheet), will allow you to use it the xl() function as usual to get a pandas DataFrame:

df = xl("A1:B2", headers=True)

But it will allow you to also get a NumPy Array:

arr = xl("A1:B2", convert=np.array)

or a list-of-lists:

mylist = xl("A1:B2", convert=list)

It’s just a few lines of code but this isn’t available right now in Python in Excel, which means code duplication and copy/pasting across workbooks.

A better alternative than “an alternative to the Excel formula language”

So I was asking myself: “Are there really no better alternatives than adding Python to improve the analytics capabilities of Excel’s formula language”? I think there is: by improving the native Excel formula language. And in recent years, that’s exactly what Microsoft has been doing. It all started with the dynamic arrays that first appeared in 2018 and that opened the door for powerful formulas like =UNIQUE() or =SORT(). Most recently, Microsoft added the following new functions:

=PIVOT()
=GROUPBY()

Wait a minute, that’s half of the reasons why I would create a pandas DataFrame in Excel, so instead of doing df.pivot_table() and df.groupby(), I can now simply stay within native Excel and get proper Excel formulas without all the clunkiness of Python in Excel.

What I find interesting though is the multiline editing experience of PY cells. Why not give the native Excel formula language an upgrade, so I don’t have to write LET expressions such as the following one (that I still find very hard to read):

=LET(x, 1, y, 2, x + y)

Instead, allow me to write it like so?

let x = 1
let y = 2
x + y

Also, why not turn Excel tables into a native Excel DataFrame? Give them attributes instead of sticking to the functional approach and you’d be looking at something like this:

=MyTable[#All].GROUPBY(...)

Lot’s of possibilities to integrate the pandas functionality in a way that feels more native to Excel!

What are my plans with xlwings?

UPDATE: xlwings Lite, my answer to Python in Excel, is now available in the official Excel add-in store for free.

One of the most popular ways of using xlwings has always been to have a Jupyter notebook and Excel running side-by-side. The caveat is that Python is required to be installed locally. That’s an issue, because Python is hard to deploy and maintain with the exact packages you need. There’s also increasing pressure from IT security: they don’t like a full Python installation on an end-user’s machine.

One solution to the problem is xlwings Server, which is self-hosted and runs everywhere where Python runs. However, the use case for xlwings Server is that of a developer, who writes macros and UDFs in Python on the backend. The end-user will access it in the form of a modern web-based Office.js add-in without having to know anything about Python. So it’s not thought for interactive use by the end-user.

That’s why I have started to work on “xlwings Lite”, which will be based on Web-Assembly (WASM). Web-Assembly basically allows you to run Python natively in the browser, while Office.js add-ins run a browser inside Excel. So I “only” need to connect Python running on WASM with the Excel object model. Pyodide, which is a Python distribution based on WASM, will help me achieve this.

While it’s likely going to take me a few more months to implement this, what I can tell from an early proof-of-concept is that it will have these characteristics:

Supports true custom functions (aka user-defined functions “UDFs”)
Gives you access to the Excel object model
Supports Matplolib/Seaborn plots
Supports web API calls
It will allow to install your own packages
Privacy first: everything will run on the user’s machine
It’s fast
It’ll support Win, macOS, and Excel on the Web from day 1
You’ll be able to upload external files without Power Query
It’ll work on the permanent versions of Office like Office 2021, not just Microsoft 365
To get it, you will simply need to install an add-in from the Excel add-in store

Basically, I want to build Office Scripts, but with Python instead of JavaScript and running on the user’s machine instead of in the Microsoft cloud.

Conclusion

There’s definitely a place for Python in Excel, but integrating a classic Jupyter notebook instead of placing the cells in the Excel sheet itself would have been much better, in my opinion.

Don’t forget that Python in Excel is still in preview as of today, so we’re likely to get improvements and fewer restrictions in the future.

A New Way to Automate Google Sheets with Python

2022-05-20T00:00:00+00:00

It’s been possible to automate Google Sheets via the Google Sheets API for many years and there are multiple Python packages available that wrap this API: there’s gspread, pygsheets, ezsheets, and Google’s own google-api-python-client. All these packages talk from Python to Google Sheets and require you to create a Google Cloud Platform project, download a credentials file, and potentially share your spreadsheet with the user from the credentials file. This can be a relatively painful process, especially if you’re new to Google’s Cloud Console. Nevertheless, the Sheets API approach is the right solution for many use cases: for example, dumping data into Google Sheets for reporting purposes or using Google Sheets as a database in your Python program.

I, however, was missing the Google Apps Script experience in Python, i.e., I wanted to develop interactive spreadsheet apps directly in Google Sheets. So I went ahead and added Google Sheets support to xlwings: instead of using the Sheets API to talk from Python to Google Sheets, xlwings goes the other way around and talks from Google Sheets to Python. The whole thing is really just a web app with a Google Sheets frontend (yes, it works with your favorite web framework!). Apart from sparing you the frontend work, this approach comes with a couple of nice features, such as:

Ease of use: no need for a Google Cloud Platform project, a credentials file, or sharing with a specific user
Authorization: leverage the Google identity and restrict certain functionality to specific users or groups
Scheduler: run your Python scripts automatically (e.g. every hour) via Google Sheets Triggers
Security: no need to store credentials, proprietary source code, or other sensitive data in Google Sheets
API: you’re not limited by the Sheets API, but can use everything that the Google Apps Script API offers

Before walking you through these points in a little more detail, let me reiterate that xlwings is not a replacement for the Sheets API approach (in fact, I am planning to wrap the Sheets API with xlwings, too)! Rather, it’s an alternative that allows you to create new types of spreadsheet applications that were previously only possible by writing JavaScript code. You could also look at it as a competitor to low-code tools for building internal apps (Retool, Internal, etc.). As a side benefit, xlwings code will also work with Microsoft Excel across Windows, macOS, and the web.

Note that Google Sheets support is part of xlwings PRO: it’s free for non-commercial use but requires a paid plan for commercial use. And finally: this post stays high-level, so if you want more details, check out the sample GitHub repos that I will link to in this post, have a look at the corresponding xlwings docs, or check my recent webinar. By the way, the screenshot of the GitHub issues dashboard is taken from this repo.

How does the traditional Sheets API work?
The xlwings quickstart for Google Sheets
Authentication and authorization
Security
Run your code via buttons, custom menus, and triggers
DataFrames and plots
Production deployment
Roadmap
Conclusion

How does the traditional Sheets API work?

Since all of the currently available Python packages such as gspread build on the Sheets API, their pre-requisites all look the same:

In the Google Cloud Console:
- Create a Google Cloud Platform project
- Enable the correct APIs
- Create the proper credentials and download them as JSON file
You might need to share your spreadsheet with the user from the created credentials
Run your Python code from outside Google Sheets, e.g., by running it from the command line on your laptop or server

There are lots of use cases where this approach works perfectly fine: running a cron job on a server to regularly push data into your sheet is a popular one. There are, however a few limitations with the Sheets API:

You can only automate what the Sheets API exposes: for example, inserting a picture on top of the cells isn’t supported
While cron jobs work great, you can’t have business users run your code ad-hoc without building some kind of app for them
Not everybody in your Google Workspace organization has the right to set up a Google Cloud Project
You need to make sure that the credentials file is properly secured

In the next section, we’ll see how xlwings works and how it ships around these issues.

The xlwings quickstart for Google Sheets

xlwings for Google Sheets is really just a web app that uses Google Sheets as the frontend, thereby saving you from having to deal with HTML, CSS, and JavaScript. It doesn’t require you to set up a Google Cloud Project, and boils down to just two steps:

Copy/paste the xlwings JavaScript module into the Apps Script editor
Run the Python web server with your xlwings code

The only caveat is that Google Sheets needs to be able to reach your web app:

For development, you can use a cloud-based IDE such as GitPod or GitHub Codespaces or—if you prefer to work on your local laptop—expose the port of your web server with a tool like ngrok.
For production, there isn’t anything special to consider as long as your server/cloud is reachable from Google Sheets.

To get up and running in literally 2 minutes, head over to the quickstart project on GitHub and hit the Gitpod button in the README. Then, in your Google Sheet, go to Extensions > Apps Script and paste the code you’ll find under js/xlwings_google.js. That’s it!

Hit the Save button in the Apps Script editor before clicking on the Run button to call the hello function: this will write Hello xlwings! into cell A1 of your spreadsheet, and clicking it again will change it to Bye xlwings!. Note that if you do this the very first time, Google will ask you to authorize the script.

How does this work? The hello function looks like this (part of the JavaScript code that we pasted into the Apps Script editor):

function hello() {
    runPython("https://your-url/hello", { apiKey: "your-api-key" });
}

The runPython call requires the URL of your Python backend for that specific function and an API key. To see the Python code that we’re calling, open app/main.py where you’ll see the corresponding endpoint:

@app.post("/hello")
def hello(data: dict = Body):
    # Instantiate a Book object with the deserialized request body
    book = xw.Book(json=data)

    # Use xlwings as usual
    sheet = book.sheets[0]
    if sheet["A1"].value == "Hello xlwings!":
        sheet["A1"].value = "Bye xlwings!"
    else:
        sheet["A1"].value = "Hello xlwings!"

    # Pass the following back as the response
    return book.json()

Note: I am using FastAPI in all of the sample repos, but you could use any other web framework like Django or Flask.

If you’ve used xlwings with Excel before, you’ll feel right at home, as you can use the same syntax. You can see the full code of a minimal quickstart sample either by looking at the repo on GitHub or by running the following command locally (requires xlwings to be installed): xlwings quickstart <projectname> --fastapi. Note that you can also use the xlwings CLI to copy the Google Apps Script module: xlwings copy gs.

If this was a bit too fast, watch this video where I’ll walk you through every step:

Now that we’ve played around with a quickstart project, let’s have a look into a few of the features that xlwings offers in connection with Google Sheets.

Authentication and authorization

Authentication is one of the most critical parts of every app and is usually hard to get right. Google Sheets makes this part embarrassingly simple: replace the API key from the quickstart sample with ScriptApp.getOAuthToken(), which allows the backend to verify the current user. This lets you:

Remove the sensitive API key from the Apps Script module
Authenticate everybody from your Workspace domain automatically
Authorize users for certain actions: for example, you can restrict write access to your database to a few users, while giving everybody read access

To get a better idea, have a look at the following sample repo:

https://github.com/xlwings/xlwings-googlesheets-fastapi-auth

It shows you:

How to set up authentication
How to get the user object
How to perform authorization (using FastAPI’s scope-based approach)

How exactly you’re authorizing your users is up to you: you could use an identity service like Active Directory or Okta or define groups as environment variables. The sample repo shows you how to verify group membership directly against your Google Workspace—this, however, requires you to set up a Google Cloud Project, see the repo’s README for all the details.

Related to authentication/authorization is security in general, which we’ll look at next.

Security

Sharing a Google Sheet is as easy as typing in an email address, but that also means that you can share it accidentally with the wrong person. To minimize the damage, xlwings makes it easy to keep sensitive data out of your spreadsheet:

Credentials for databases, APIs, etc.
Other sensitive data like client data
Business logic (your Python source code)

The Apps Script module will only contain non-sensitive information, as long as you’re using ScriptApp.getOAuthToken() as the apiToken (see the previous section). If your spreadsheet gets into the wrong hands, they will only be able to see the spreadsheet, but they won’t be able to run your Python code. And since you don’t have any credentials in the spreadsheet, you won’t need to rotate them either.

Importantly, credentials can be properly treated as secrets on your backend, using whatever means your provider offers. You can also control who should have access to the Python code (via Git version control, of course).

Now that your app is secure, let’s have a look at different ways of running your code.

Run your code via buttons, custom menus, and triggers

xlwings can take advantage of the same possibilities that you have when running Google Apps Script code:

Buttons: in Google Sheets, you can click on Insert > Drawing, draw something that looks like a button (e.g., a rounded rectangle), and hit Save and Close. Then, right-click on the button so the three dots appear, click on them and select Assign script, then write the name of the function, in the case of the quickstart sample, this would be hello.
Custom Menus: with a few lines of JavaScript code, you can run your function from the menu, see the official docs.
Triggers: in the Apps Script editor, click on the alarm clock icon in the sidebar on the left. Then, at the bottom-right, click on the blue Add Trigger button: this will allow you to run your function periodically or based on a spreadsheet action like the open event.

If you are into scientific computing or machine learning, chances are that your buttons and menus will run some pandas code. Let’s have a closer look at how you work with the scientific Python stack in the next section!

DataFrames and plots

NumPy arrays, pandas DataFrames, and Matplotlib are at the very heart of scientific computing with Python. Accordingly, xlwings supports all of them out of the box. Plotly is also supported, and every other plotting library will also work, as long as the plots can be exported to a picture. The screenshot at the beginning of this section is taken from https://github.com/xlwings/xlwings-googlesheets-pandas-plots and as you will see there, writing a DataFrame df to Google Sheets boils down to:

mysheet["A1"].value = df

Using a Matplotlib plot—as generated by pandas—looks something like this:

ax = df.plot(figsize=(12, 8))
mysheet.pictures.add(image=ax.get_figure(), anchor=sheet["E9"])

For more details, have a look at the repo.

In the next section, we’ll have a look at how you can deploy these sample apps into production.

Production deployment

The number one issue with xlwings has always been the deployment of Python—which is the reason I love this part so much: throw a Docker image or Dockerfile at your favorite service for running Docker containers and you’re done! There are a ton of amazing services in the cloud or under your desk, and you can choose the one that fits your requirements. For more information and a few suggestions with regard to services, see the xlwings docs.

Most of the sample repos mentioned in this post have a Dockerfile included that makes it straightforward to deploy them directly from GitHub. An even easier option is to click the Deploy to Render button (this will deploy it to their free tier), see e.g., https://github.com/xlwings/xlwings-googlesheets-pandas-plots.

Let’s conclude this post with a quick review of the roadmap!

Roadmap

It’s still early stage for Google Sheets support in xlwings, however, with pandas DataFrames and Python plots alone, you’ll be able to build amazing things! The next steps will be:

Add support for custom functions (a.k.a. user-defined functions)
Add support for the full xlwings API, including charts, named ranges, etc.
Making everything more efficient

The exact order will be dictated by you, the user, of course.

Conclusion

The new Google Sheets support in xlwings allows you to build tools in Python that you could previously only build in Google Apps Script, i.e., JavaScript. xlwings allows you to leverage Google’s built-in authentication, you can make the spreadsheet tools secure by keeping the sensitive data out of your spreadsheet, and you have total freedom with regard to the web framework and infrastructure for your backend.

What Makes Excel’s Lambda Functions so Awesome (and what doesn’t)?

2021-02-04T00:00:00+00:00

In December 2020, Microsoft announced Lambda Functions as a new Excel feature. Lambda functions are an exciting way to define custom functions (a.k.a. user-defined functions) by only relying on Excel formulas, without having to use VBA or writing an add-in. In this blog post, we’re going to see how Lambda Functions work and why they are so awesome. It’s not all roses though: we’ll also look at the current limitations of Lambda Functions and how to solve some of them. Currently, Lambda Functions are only available to Microsoft 365 subscribers on the Beta Channel. To see if you can already use them, start typing =LAMBDA in a cell and if Excel suggests the function name, you are all set. Would you rather lean back and watch the gist of it in a video? Here you go:

Why Do We Need Lambda Functions?
Create your First Lambda Function!
How Do Lambda Functions Work?
Why are Lambda Functions so Awesome
Limitations with Lambda Functions
Managing Lambda Functions Across Workbooks
Keeping Track of Your Lambda Functions
Conclusion

Why Do We Need Lambda Functions?

Let’s say you have an Excel workbook with a list of temperatures in Fahrenheit and Kelvin that you want to transform into Celsius. After looking up the formulas on Wikipedia, chances are you end up writing the following cell formula into cell C2, before copying it to the cells below:

This approach has two issues:

If it wasn’t for the header in cell C1, you wouldn’t necessarily know what that formula is doing, which makes it harder to review and spot errors
If you make an error in your formula, let’s say you use 9/5 instead of 5/9 in the Fahrenheit conversion, you need to search for all formulas across your entire workbook and make sure that every single one is corrected

The new Lambda Functions solve both of these issues, let’s see how!

Create your First Lambda Function!

Let’s start by defining and using a Lambda Function so we’ll get a feeling for how they work before we’ll get to a more detailed explanation in the next section. In Excel, go to Formulas > Name Manager and click on New (macOS: Formulas > Define Name). Fill in the following values; see screenshot:

Name: ToCelsius
Refers to: =LAMBDA(degrees, source, IF(source = "fahrenheit", (degrees-32) * (5/9), IF(source = "kelvin", degrees - 273.15, NA())))
Comments: Feel free to add comments to explain the parameters and formula, but this is only available on Windows, not macOS.

On macOS, the two fields appear right after clicking on Define Name and have the verbose titles Enter a name for the data range instead of Name and Select the range of cells instead of Refers to.

Save by clicking on OK and now you’re able to use your new formula like this:

That’s quite a bit easier to understand, isn’t it? And the best part is that you can make changes to the formula in a single place (via the Name Manager) and it’s fixed everywhere automatically.

But how do Lambda Functions work exactly? The next section has the details.

How Do Lambda Functions Work?

Lambda is a new function that accepts the following parameters:

=LAMBDA(parameter1, parameter2, ..., calculation)

The first n parameters are the parameters of your custom function (you can use a max. of 253 parameters), in our case we use two: degrees and source. The last parameter is the actual custom function, here called calculation. Being able to refer to explicit parameter names instead of cell references makes the formula already quite a bit easier to understand. Here is, for example, the Fahrenheit conversion:

Without Lambda Functions: IF(B2 = "fahrenheit", (A2-32) * (5/9)
In the context of a Lambda Function: IF(source = "fahrenheit", (degrees-32) * (5/9)

If you like, you could try out a Lambda Function directly in an Excel cell like any other formula, but you will have to call it in the very same cell by adding parentheses with the parameters at the end, otherwise, you would get a #CALC! error. Accordingly, to test our formula without setting it up in the Name Manager, you could write the following in a cell (note the (20, "fahrenheit") at the end of the formula which is calling the Lambda Function with these two parameters):

=LAMBDA(degrees, source, IF(source = "fahrenheit", (degrees-32) * (5/9), IF(source = "kelvin", degrees - 273.15, NA())))(20, "fahrenheit")

This can be helpful for testing, but you can’t reuse your formula until you assign a name to it in the Name Manager. Now that we have seen how you define and use Lambda Functions, let’s summarize their advantages.

Why are Lambda Functions so Awesome

As a summary, here is why Lambda Functions are such a big deal:

Users don’t need to learn another language like VBA to write custom functions.
This means that you can define custom functions without having to save your Excel workbook in the macro-enabled xlsm file—not having to click on the Enable Macros button will make everybody’s life easier.
Using Lambda Function means that you change formulas in exactly one spot and it’s going to be changed everywhere you use it. This is a massive improvement in terms of making your workbook less error-prone.
Instead of long, difficult-to-guess formulas, you can use short, descriptive names for your custom functions which makes it much easier to understand your workbook—especially for others!

With the advantages covered, let’s not forget about the current limitations and downsides of Lambda Functions.

Limitations with Lambda Functions

To be fair: Lambda Functions are brand new, so it’s expected that there are still a few rough edges. Microsoft also explicitly mentions on their announcement that this initial release is just the beginning and that they’ll look into improving the functionality based on user feedback going forward. Accordingly, the following points should be understood as constructive feedback:

To start with the obvious: everybody you work with will need the latest version of Microsoft Excel to get access to Lambda Functions.
Lambda Functions are hidden away: it may not be instantly obvious to a user why two seemingly same workbooks can yield different values. This could happen if you define the ToCelsius formula in one workbook with an error and in another one correctly.
This is related to the previous point: Lambda Function can’t be reused across different workbooks which is—in my view—their biggest limitation (but I’ll show you a workaround below).
All parameters of a Lambda Function are optional, which becomes an issue if you just want to provide the 7th parameter in a function with 10 parameters: =MyFunction(,,,,,,"some argument",,,). Good luck figuring out which argument that is! It would help, if Lambda Functions would accept named parameters similar to how VBA does it, so you could write instead =MyFunction(some_parameter:="some argument") without having to rely on an argument’s position.
With Lambda Functions, you don’t get a tooltip that shows you which argument is which, something that you are used to from the built-in functions. I really hope this will be added later on so they will become easier to use.
It’s very hard to type in a formula into the Name Manager (and macOS doesn’t even have a proper Name Manager!). Even though our ToCelsius function is very simple, even just looking for a typo is nothing else than awkward. So please, Microsoft, give us a proper editor with (a) multiple lines, (b) a monospaced font, and (c) syntax highlighting to edit these functions (something that would be useful for normal functions, too, by the way). Seeing the formula capped to 40 characters in the Name Manager is no fun:
```
  =LAMBDA(degrees, source, IF(source = "fa
```
Instead, a proper formula editor would show it like this, which gives you a chance to actually understand what’s going on:
```
  =LAMBDA(
      degrees,
      source,
      IF(
         source = "fahrenheit",
         (degrees-32) * (5/9),
         IF(
            source = "kelvin",
            degrees - 273.15,
            NA()
         )
      )
  )
```

Some of these issues have a solution: the next section continues by showing you how to work around the inability of sharing Lambda Functions across workbooks.

Managing Lambda Functions Across Workbooks

I am a Python guy, so this is the tool I use to solve pretty much all my issues in life, including the one that my Lambda Functions from one workbook don’t magically appear in all my other workbooks where I want them. I won’t be covering how to get started with Python in this blog post, but there are enough resources on the web in case you are a Python newbie (downloading the Anaconda or WinPython distribution is usually a good place to start—if you use a different distro, just make sure to run pip install xlwings to install the xlwings package). At a later point, I might also consider turning this into a little command line application to make it easier to use. Anyhow, for now, create the following file and call it lambda_report.py:

# lambda_report.py
from pathlib import Path
import xlwings as xw

# TODO: adjust the DIRECTORY path
DIRECTORY = r'C:\Users\felix\Desktop\lambda'

app = xw.App(visible=False)
for path in Path(DIRECTORY).rglob('[!~$]*.xls*'):
    book = app.books.open(path)
    print(f'------ {path} ------')
    for name in book.names:
        refers_to = name.refers_to.replace('_xlfn.', '').replace('_xlpm.', '')
        lambda_functions = []
        if refers_to.lower().startswith('=lambda'):
            lambda_functions.append(f'{name.name}: {refers_to}')
        if lambda_functions:
            for func in lambda_functions:
                print(func)
    print()
    book.close()
app.quit()

Make sure to update the path for DIRECTORY and run it on your Anaconda Prompt or Command Prompt on Windows or Terminal on macOS as follows (or use your favorite Python editor):

python lambda_report.py

This will print all Lambda Functions for all Excel workbooks in the indicated DIRECTORY including its sub-directories like so (the Lambda Functions from test_book.xlsx are taken from the official documentation):

------ C:\Users\felix\Desktop\lambda\temperature.xlsx ------
ToCelsius: =LAMBDA(degrees,source, IF(source = "fahrenheit", (degrees-32) * (5/9), IF(source = "kelvin", degrees - 273.15, NA())))

------ C:\Users\felix\Desktop\lambda\subfolder\test_book.xlsx ------
Hypotenuse: =LAMBDA(a, b, SQRT((a^2+b^2)))
CountWords: =LAMBDA(text, LEN(TRIM(text)) - LEN(SUBSTITUTE(TRIM(text), " ", "")) + 1)
ThanksgivingDate: =LAMBDA(year, TEXT(DATE(year, 11, CHOOSE(WEEKDAY(DATE(year, 11, 1)), 26, 25, 24, 23, 22, 28, 27)), "mm/dd/yyyy"))

Now that you have your report and know which workbooks contain which Lambda Functions, you may want to add or correct a couple of Lambda Functions across a bunch of workbooks. Here is how this works: create a new Python file called add_lambdas.py:

# add_lambdas.py
from pathlib import Path
import xlwings as xw

# TODO: adjust the DIRECTORY path and the LAMBDAS you want to add
DIRECTORY = r'C:\Users\felix\Desktop\lambda'

LAMBDAS = (
    ('Hypotenuse', '=LAMBDA(a, b, SQRT((a^2+b^2)))'),
    ('CountWords', '=LAMBDA(text, LEN(TRIM(text)) - LEN(SUBSTITUTE(TRIM(text), " ", "")) + 1)')
)

app = xw.App(visible=False)
for path in Path(DIRECTORY).rglob('[!~$]*.xls*'):
    print(f'Adding Lambda Functions to: {path}')
    book = app.books.open(path)
    for func in LAMBDAS:
        name, refers_to = func[0], func[1]
        if name in [n.name for n in book.names]:
            book.names[func[0]].delete()
        book.names.add(name=name, refers_to=refers_to)
    book.save()
    book.close()
app.quit()

Again, make sure to update DIRECTORY and LAMBDAS to suit your needs, then run the script like this:

python add_lambdas.py

This will add the two Lambda Functions Hypotenuse and CountWords to all your Excel files in the specified DIRECTORY and its subdirectories—if the name already exists, it will be replaced.

Now that we’ve seen an easy way how to manage your Lambda Functions across workbooks, let’s learn about how we can keep track of them and make sure that nobody messes with our Lambada Functions!

Keeping Track of Your Lambda Functions

When you use Lambda Functions, you have your critical business logic sitting in single-line formulas of which you can only see about 40 characters at a time in the Name Manager. This doesn’t exactly make it easy to review your formulas. If you make a change to a Lambda Function, it’s easy to introduce errors (especially because Excel constantly thinks you want to reference cells in the Refers To section), so you better watch your back!

To keep the upper hand in software development, professional coders use a version control system like Git, usually through a provider like GitHub, GitLab, or BitBucket. Git keeps track of code changes and allows team members to perform a peer review by highlighting changes (think of it as the track changes functionality in Word). Git, however, has two issues when you use it with Excel files:

Git doesn’t understand Excel files and isn’t able to show you their contents or changes
Git has a steep learning curve that doesn’t really appeal to spreadsheet users, as the vast majority of them are business users, not software developers

Still, we believe that version control is not something that you should ignore just because you use Excel, which is the reason why we built xltrail: xltrail is a Git system that is easy to use for business users via drag and drop and knows how to deal with Excel files. Let’s see how xltrail keeps track of Lambda Functions by looking at the Excel file of the first sample of this blog post:

Clicking on Sheet1 will give you the same view as the very first screenshot in this blog post, but you will see the formulas:

But let’s not get sidetracked here—after all, I wanted to show you how Lambda Functions are tracked. When click on DefinedNames instead of Sheet1, you will see all your Lambda Functions:

Now, the right sidebar doesn’t just give you access to past versions but most importantly, you can click on Show changes to see what changed:

This allows you to verify that I have indeed properly fixed the formula for the Fahrenheit conversion (red is the previous version and green is the current version) and you know why you’re getting wrong results when you use an older version of this workbook. If you want to try this out on your own, sign up for a free trial!

Conclusion

My goal with this blog post was to show you what Lambda Functions are, why they are great and how you use them in practice. Unlike many other blog posts though, I didn’t leave it at this but moved on by looking at the challenges that come with Lambda Functions, my main critique being that they are hard to share and keep track of—especially since the Name Manager isn’t a good formula editor.

Given how ecstatic Lambda Functions have been received by the community, I am pretty sure that Microsoft will make them better over time, so I do hope that we will get a proper formula editor and a way to share Lambda Functions between workbooks. In the meantime, you can always use my short Python scripts to keep your Lambda Functions under control with a minimum of fuss!

5 Python Libraries for Reporting and Factsheets

2020-06-03T00:00:00+00:00

Python is a popular tool for all kind of automation needs and therefore a great candidate for your reporting tasks. There is a wealth of techniques and libraries available and we’re going to introduce five popular options here. After reading this blog post, you should be able to pick the right library for your next reporting project according to your needs and skill set.

Overview
Pandas
xlwings
Plotly Dash
Datapane
ReportLab
Conclusion

Overview

Before we begin, here is a high level comparison of the libraries presented in this post:

Library	Technology	Summary
Pandas + HTML	HTML	You can generate beautiful reports in the form of static web pages if you know your way around HTML + CSS. The HTML report can also be turned into a PDF for printing.
Pandas + Excel	Excel	This is a great option if the report has to be in Excel. It can be run on a server where Excel is not installed, i.e. it’s an ideal candidate for a “download to Excel” button in a web app. The Excel file can be exported to PDF.
xlwings	Excel	xlwings allows the use of an Excel template so the formatting can be done by users without coding skills. It requires, however, an installation of Excel so it’s a good option when the report can be generated on a desktop, e.g. for ad-hoc reports. The Excel file can be exported to PDF.
Plotly Dash	HTML	Dash allows you to easily spin up a great looking web dashboard that is interactive without having to write any JavaScript code. If formatted properly, it can be used as a source for PDFs, too. Like Pandas + HTML, it requires good HTML + CSS skills to make it look the way you want.
Datapane	HTML	Datapane allows you to create HTML reports with interactive elements. It also offers a hosted solution so end users can change the input parameters that are used to create these reports.
ReportLab	PDF	ReportLab creates direct PDF files without going through HTML or Excel first. It’s very fast and powerful but comes with a steep learning curve. Used by Wikipedia for their PDF export.

Pandas

I am probably not exaggerating when I claim that almost all reporting in Python starts with Pandas. It’s incredibly easy to create Pandas DataFrames with data from databases, Excel and csv files or json responses from a web API. Once you have the raw data in a DataFrame, it only requires a few lines of code to clean the data and slice & dice it into a digestible form for reporting. Accordingly, Pandas will be used in all sections of this blog post, but we’ll start by leveraging the built-in capabilities that Pandas offers for reports in Excel and HTML format.

Pandas + Excel

Required libraries: pandas, xlsxwriter

If you want to do something slightly more sophisticated than just dumping a DataFrame into an Excel spreadsheet, I found that Pandas and XlsxWriter is the easiest combination, but others may prefer OpenPyXL. In that case you should be able to easily adopt this snippet by replacing engine='xlsxwriter' with engine='openpyxl' and changing the book/sheet syntax so it works with OpenPyXL:

import pandas as pd
import numpy as np

# Sample DataFrame
df = pd.DataFrame(np.random.randn(5, 4), columns=['one', 'two', 'three', 'four'],
                  index=['a', 'b', 'c', 'd', 'e'])

# Dump Pandas DataFrame to Excel sheet
writer = pd.ExcelWriter('myreport.xlsx', engine='xlsxwriter')
df.to_excel(writer, sheet_name='Sheet1', startrow=2)

# Get book and sheet objects for futher manipulation below
book = writer.book
sheet = writer.sheets['Sheet1']

# Title
bold = book.add_format({'bold': True, 'size': 24})
sheet.write('A1', 'My Report', bold)

# Color negative values in the DataFrame in red
format1 = book.add_format({'font_color': '#E93423'})
sheet.conditional_format('B4:E8', {'type': 'cell', 'criteria': '<=', 'value': 0, 'format': format1})

# Chart
chart = book.add_chart({'type': 'column'})
chart.add_series({'values': '=Sheet1!B4:B8', 'name': '=Sheet1!B3', 'categories': '=Sheet1!$A$4:$A$8'})
chart.add_series({'values': '=Sheet1!C4:C8', 'name': '=Sheet1!C3'})
chart.add_series({'values': '=Sheet1!D4:D8', 'name': '=Sheet1!D3'})
chart.add_series({'values': '=Sheet1!E4:E8', 'name': '=Sheet1!E3'})
sheet.insert_chart('A10', chart)

writer.save()

Running this will produce the following report:

Of course, we could now go back to the script and add more code to style it a bit nicer, but I leave this as an exercise to the reader…

Pandas + HTML

Required libraries: pandas, jinja2

Creating an HTML report with pandas works similar to what’ve just done with Excel: If you want a tiny bit more than just dumping a DataFrame as a raw HTML table, then you’re best off by combining Pandas with a templating engine like Jinja:

First, let’s create a file called template.html:

<html>
<head>
    <style>
        * {
            font-family: sans-serif;
        }
        body {
            padding: 20px;
        }
        table {
            border-collapse: collapse;
            text-align: right;
        }
        table tr {
            border-bottom: 1px solid
        }
        table th, table td {
            padding: 10px 20px;
        }
    </style>
</head>
<body>

<h1>My Report</h1>

{{ my_table }}

<img src='plot.svg' width="600">

</body>
</html>

Then, in the same directory, let’s run the following Python script that will create our HTML report:

import pandas as pd
import numpy as np
import jinja2

# Sample DataFrame
df = pd.DataFrame(np.random.randn(5, 4), columns=['one', 'two', 'three', 'four'],
                  index=['a', 'b', 'c', 'd', 'e'])

# See: https://pandas.pydata.org/pandas-docs/stable/user_guide/style.html#Building-styles
def color_negative_red(val):
    color = 'red' if val < 0 else 'black'
    return f'color: {color}'

styler = df.style.applymap(color_negative_red)

# Template handling
env = jinja2.Environment(loader=jinja2.FileSystemLoader(searchpath=''))
template = env.get_template('template.html')
html = template.render(my_table=styler.render())

# Plot
ax = df.plot.bar()
fig = ax.get_figure()
fig.savefig('plot.svg')

# Write the HTML file
with open('report.html', 'w') as f:
    f.write(html)

The result is a nice looking HTML report that could also be printed as a PDF by using something like WeasyPrint:

Note that for such an easy example, you wouldn’t necessarily need to use a Jinja template. But when things start to become more complex, it’ll definitely come in very handy.

xlwings

xlwings allows you to program and automate Excel with Python instead of VBA. The difference to XlsxWriter or OpenPyXL (used above in the Pandas section) is the following: XlsxWriter and OpenPyXL write Excel files directly on disk. They work wherever Python works and don’t require an installation of Microsoft Excel. xlwings, on the other hand, can write, read and edit Excel files via the Excel application, i.e. a local installation of Microsoft Excel is required. xlwings also allows you to create macros and user-defined functions in Python rather than in VBA, but for reporting purposes, we won’t really need that.

While XlsxWriter/OpenPyXL are the best choice if you need to produce reports in a scalable way on your Linux web server, xlwings does have the advantage that it can edit pre-formatted Excel files without losing or destroying anything. OpenPyXL on the other hand (the only writer library with xlsx editing capabilities) will drop some formatting and sometimes leads to Excel raising errors during further manual editing.

xlwings (Open Source)

Replicating the sample we had under Pandas is easy enough with the open-source version of xlwings:

import xlwings as xw
import pandas as pd
import numpy as np

# Open a template file
wb = xw.Book('mytemplate.xlsx')

# Sample DataFrame
df = pd.DataFrame(np.random.randn(5, 4), columns=['one', 'two', 'three', 'four'],
                  index=['a', 'b', 'c', 'd', 'e'])

# Assign data to cells
wb.sheets[0]['A1'].value = 'My Report'
wb.sheets[0]['A3'].value = df

# Save under a new file name
wb.save('myreport.xlsx')

Running this will produce the following report:

So where does all the formatting come from? The formatting is done directly in the Excel template before running the script. This means that instead of having to program tens of lines of code to format a single cell with the proper font, colors and borders, I can just make a few clicks in Excel. xlwings then merely opens the template file and inserts the values.

This allows us to create a good looking report in your corporate design very fast. The best part is that the Python developer doesn’t necessarily have to do the formatting but can leave it to the business user who owns the report.

Note that you could instruct xlwings to run the report in a separate and hidden instance of Excel so it doesn’t interfere with your other work.

xlwings PRO

The Pandas + Excel as well as the xlwings (open source) sample both have a few issues:

If, for example, you insert a few rows below the title, you will have to adjust the cell references accordingly in the Python code. Using named ranges could help but they have other limitations (like the one mentioned at the end of this list).
The number of rows in the table might be dynamic. This leads to two issues: (a) data rows might not be formatted consistently and (b) content below the table might get overwritten if the table is too long.
Placing the same value in a lot of different cells (e.g. a date in the source note of every table or chart) will cause duplicated code or unnecessary loops.

To fix these issues, xlwings PRO comes with a dedicated reports package:

Separation of code and design: Users without coding skills can change the template on their own without having to touch the Python code.
Template variables: Python variables (between double curly braces) can be directly used in cells , e.g. {{ title }}. They act as placeholders that will be replaced by the values of the variables.
Frames for dynamic tables: Frames are vertical containers that dynamically align and style tables that have a variable number of rows. To see how Frames work, have a look at the documentation.

You can get a free trial for xlwings PRO here. When using the xlwings PRO reports package, your code simplifies to the following:

import pandas as pd
import numpy as np
from xlwings.pro.reports import create_report  # part of xlwings PRO

# Sample DataFrame
df = pd.DataFrame(np.random.randn(5, 4), columns=['one', 'two', 'three', 'four'],
                  index=['a', 'b', 'c', 'd', 'e'])

# Create the report by passing in all variables as kwargs
wb = create_report('mytemplate.xlsx',
                   'myreport.xlsx',
                   title='My Report',
                   df=df)

All that’s left is to create a template with the placeholders for title and df:

Running the script will produce the same report that we generated with the open source version of xlwings above. The beauty of this approach is that there are no hard coded cell references anymore in your Python code. This means that the person who is responsible for the layout can move the placeholders around and change the fonts and colors without having to bug the Python developer anymore.

Plotly Dash

Required libraries: pandas, dash

Plotly is best known for their beautiful and open-source JavaScript charting library which builds the core of Chart Studio, a platform for collaboratively designing charts (no coding required).

To create a report though, we’re using their latest product Plotly Dash, an open-source framework that allows the creation of interactive web dashboards with Python only (no need to write JavaScript code). Plotly Dash is also available as Enterprise plan.

How it works is best explained by looking at some code, adopted with minimal changes from the official getting started guide:

import pandas as pd
import dash
import dash_core_components as dcc
import dash_html_components as html
from dash.dependencies import Input, Output

# Sample DataFrame
df = pd.read_csv('https://raw.githubusercontent.com/plotly/datasets/master/gapminderDataFiveYear.csv')

# Dash app - The CSS code is pulled in from an external file
app = dash.Dash(__name__, external_stylesheets=['https://codepen.io/chriddyp/pen/bWLwgP.css'])

# This defines the HTML layout
app.layout = html.Div([
    html.H1('My Report'),
    dcc.Graph(id='graph-with-slider'),
    dcc.Slider(
        id='year-slider',
        min=df['year'].min(),
        max=df['year'].max(),
        value=df['year'].min(),
        marks={str(year): str(year) for year in df['year'].unique()},
        step=None
    )
])

# This code runs every time the slider below the chart is changed
@app.callback(Output('graph-with-slider', 'figure'), [Input('year-slider', 'value')])
def update_figure(selected_year):
    filtered_df = df[df.year == selected_year]
    traces = []
    for i in filtered_df.continent.unique():
        df_by_continent = filtered_df[filtered_df['continent'] == i]
        traces.append(dict(
            x=df_by_continent['gdpPercap'],
            y=df_by_continent['lifeExp'],
            text=df_by_continent['country'],
            mode='markers',
            opacity=0.7,
            marker={'size': 15, 'line': {'width': 0.5, 'color': 'white'}},
            name=i
        ))

    return {
        'data': traces,
        'layout': dict(
            xaxis={'type': 'log', 'title': 'GDP Per Capita', 'range': [2.3, 4.8]},
            yaxis={'title': 'Life Expectancy', 'range': [20, 90]},
            margin={'l': 40, 'b': 40, 't': 10, 'r': 10},
            legend={'x': 0, 'y': 1},
            hovermode='closest',
            transition={'duration': 500},
        )
    }

if __name__ == '__main__':
    app.run_server(debug=True)

Running this script and navigating to http://localhost:8050 in your browser will give you this dashboard:

The charts look great by default and it’s very easy to make your dashboard interactive by writing simple callback functions in Python: You can choose the year by clicking on the slider below the chart. In the background, every change to our year-slider will trigger the update_figure callback function and hence update the chart.

By arranging your documents properly, you could create an interactive web dashboard that can also act as the source for your PDF factsheet, see for example their financial factsheet demo together with it’s source code.

Alternatives to Plotly Dash

If you are looking for an alternative to Plotly Dash, make sure to check out Panel. Panel was originally developed with the support of Anaconda Inc., and is now maintained by Anaconda developers and community contributors. Unlike Plotly Dash, Panel is very inclusive and supports a wide range of plotting libraries including: Bokeh, Altair, Matplotlib and others (including also Plotly).

Datapane

Required libraries: datapane

Datapane is a framework for reporting which allows you to generate interactive reports from pandas DataFrames, Python visualisations (such as Bokeh and Altair), and Markdown. Unlike solutions such as Dash, Datapane allows you to generate standalone reports which don’t require a running Python server—but it doesn’t require any HTML coding either.

Using Datapane, you can either generate one-off reports, or deploy your Jupyter Notebook or Python script so others can generate reports dynamically by entering parameters through an automatically generated web app.

Datapane (open-source library)

Datapane’s open-source library allows you to create reports from components, such as a Table component, a Plot component, etc. These components are compatible with Python objects such as pandas DataFrames, and many visualisation libraries, such as Altair:

import datapane as dp
import pandas as pd
import altair as alt

df = pd.read_csv('https://query1.finance.yahoo.com/v7/finance/download/GOOG?period2=1585222905&interval=1mo&events=history')

chart = alt.Chart(df).encode(
    x='Date:T',
    y='Open'
).mark_line().interactive()

r = dp.Report(dp.Table(df), dp.Plot(chart))
r.save(path='report.html')

This code renders a standalone HTML document with an interactive, searchable table and plot component.

If you want to publish your report, you can login to Datapane (via $ datapane login) and use the publish method, which will give you a URL such as this which you can share or embed.

r.publish(name='my_report')

Hosted Reporting Apps

Datapane can also be used to deploy Jupyter Notebooks and Python scripts so that other people who are not familiar with Python can generate custom reports. By adding a YAML file to your folder, you can specify input parameters as well as dependencies (through pip, Docker, or local folders). Datapane also has support for managing secret variables, such as database passwords, and for storing and persisting files. Here is a sample script (stocks.py) and YAML file (stocks.yaml):

# stocks.py
import datapane as dp
import altair as alt
import yfinance as yf

dp.Params.load_defaults('./stocks.yaml')

tickers = dp.Params.get('tickers')
plot_type = dp.Params.get('plot_type')
period = dp.Params.get('period')
data = yf.download(tickers=' '.join(tickers), period=period, groupby='ticker').Close

df = data.reset_index().melt('Date', var_name='symbol', value_name='price')
base_chart = alt.Chart(df).encode(x='Date:T', y='price:Q', color='symbol').interactive()
chart = base_chart.mark_line() if plot_type == 'line' else base_chart.mark_bar()
dp.Report(dp.Plot(chart), dp.Table(df)).publish(name=f'stock_report', headline=f'Report on {" ".join(tickers)}')

# stocks.yaml
name: stock_analysis
script: stocks.py
# Script parameters
parameters:
  - name: tickers
    description: A list of tickers to plot
    type: list
    default: ['GOOG', 'MSFT', 'IBM']
  - name: period
    description: Time period to plot
    type: enum
    choices: ['1d','5d','1mo','3mo','6mo','1y','2y','5y','10y','ytd','max']
    default: '1mo'
  - name: plot_type
    type: enum
    default: line
    choices: ['bar', 'line']

## Python packages required for the script
requirements:
  - yfinance

Publishing this into a reporting app is easy as running $ datapane script deploy. For a full example see this example GitHub repository or read the docs.

ReportLab

Required libraries: pandas, reportlab

ReportLab writes PDF files directly. Most prominently, Wikipedia uses ReportLab to generate their PDF exports. One of the key strength of ReportLab is that it builds PDF reports “at incredible speeds”, to cite their homepage. Let’s have a look at some sample code for both the open-source and the commercial version!

ReportLab OpenSource

In its most basic functionality, ReportLab uses a canvas where you can place objects using a coordinate system:

from reportlab.pdfgen import canvas
c = canvas.Canvas("hello.pdf")
c.drawString(50, 800, "Hello World")
c.showPage()
c.save()

ReportLab also offers an advanced mode called PLATYPUS (Page Layout and Typography Using Scripts), which is able to define dynamic layouts based on templates at the document and page level. Within pages, Frames would then arrange Flowables (e.g. text and pictures) dynamically according to their height. Here is a very basic example of how you put PLATYPUS at work:

import pandas as pd
import numpy as np
from reportlab.pdfgen.canvas import Canvas
from reportlab.lib import colors
from reportlab.lib.styles import getSampleStyleSheet
from reportlab.lib.units import inch
from reportlab.platypus import Paragraph, Frame, Table, Spacer, TableStyle

# Sample DataFrame
df = pd.DataFrame(np.random.randn(5, 4), columns=['one', 'two', 'three', 'four'],
                  index=['a', 'b', 'c', 'd', 'e'])

# Style Table
df = df.reset_index()
df = df.rename(columns={"index": ""})
data = [df.columns.to_list()] + df.values.tolist()
table = Table(data)
table.setStyle(TableStyle([
    ('INNERGRID', (0, 0), (-1, -1), 0.25, colors.black),
    ('BOX', (0, 0), (-1, -1), 0.25, colors.black)
]))

# Components that will be passed into a Frame
story = [Paragraph("My Report", getSampleStyleSheet()['Heading1']),
         Spacer(1, 20),
         table]

# Use a Frame to dynamically align the compents and write the PDF file
c = Canvas('report.pdf')
f = Frame(inch, inch, 6 * inch, 9 * inch)
f.addFromList(story, c)
c.save()

Running this script will produce the following PDF:

ReportLab PLUS

In comparison to the open-source version of ReportLab, the most prominent features of Reportlab PLUS are

a templating language
the ability to include vector graphics

The templating language is called RML (Report Markup Language), an XML dialect. Here is a sample of how it looks like, taken directly from the official documentation:

<!DOCTYPE document SYSTEM "rml.dtd">
<document filename="example.pdf">
    <template>
        <pageTemplate id="main">
            <frame id="first" x1="72" y1="72" width="451" height="698" />
        </pageTemplate>
    </template>
    <stylesheet>
    </stylesheet>
    <story>
        <para>
            This is the "story". This is the part of the RML document where
            your text is placed.
        </para>
        <para>
            It should be enclosed in "para" and "/para" tags to turn it into
            paragraphs.
        </para>
    </story>
</document>

The idea here is that you can have any program produce such an RML document, not just Python, which can then be transformed into a PDF document by ReportLab PLUS.

Conclusion

Python offers various libraries to create professional reports and factsheets. If you are a good at HTML + CSS have a look at Plotly Dash or Panel or write your HTML documents directly with the help of the to_html method in Pandas.

If you need your report as Excel file (or if you hate CSS), Pandas + XlsxWriter/OpenPyXL or xlwings might be the right choice - you can still export your Excel document as PDF file. xlwings is the better choice if you want to split the design and code work. XlsxWriter/OpenPyxl is the better choice if it needs to be scalable and run on a server.

If you need to generate PDF files at high speed, check out ReportLab. It has a steep learning curve and requires to write quite some code but once the code has been written, it works at high speed.

Dynamic arrays in Excel with Python and xlwings

2019-07-05T00:00:00+00:00

The days of Ctrl-Shift-Enter and You can't change part of an array are finally over! Say hello to the new dynamic arrays that Microsoft has now (July 2019) rolled out to a broader user base: their Office 365 subscribers who are on the monthly channel.

To check if you are on the monthly channel, go to File > Account and see if it mentions Monthly Channel (Targeted) under About Excel. If you are not, then you’ll need to wait a bit longer (or switch to the monthly channel). Also note that dynamic arrays are not yet part of Office 2019, so you will absolutely need an Office 365 subscription.

To quickly double check if the respective update has already been installed, try if you have access to the =UNIQUE formula. If you do, you’re all set for Excel’s most revolutionary feature since the introduction of the ribbon and the xlsx format. If you don’t, go to File > Account > Office Updates and click on Update Now.

Dynamic arrays are obviously great news if you work with Excel in general, but they also make writing UDFs (User Defined Functions) in Python via xlwings a lot more powerful. This post assumes you know how xlwings UDFs work, so head over to the xlwings quickstart if you need a refresher.

Overview

What’s wrong with the legacy Ctrl-Shift-Enter array formulas?
The new dynamic array formulas and #Spill errors
xlwings UDFs and the new dynamic arrays
New built-in Excel formulas: UNIQUE etc.

Prefer to watch a video? Here it is:

What’s wrong with the legacy Ctrl-Shift-Enter array formulas?

Consider the following xlwings UDF:

import xlwings as xw
import numpy as np

@xw.func
def dynamic_array(r, c):
    return np.random.rand(int(r), int(c))

In older versions of Excel, if you write =dynamic_array(4, 3) into A1, then you would just get one value back instead of the full 4 x 3 array. To solve that, you’d have to use legacy array formulas: Select all cells for the result array (i.e. A1:C4 in the example), then, while the cells are selected, write the formula into the top left cell and finally hit Ctrl-Shift-Enter (see the xlwings docs for a step-by-step tutorial).

The biggest issue arises when you want to change the formula to =dynamic_array(5, 5): You have to first delete the entire array (you need to select the entire array to be able to do that), then select the new 5 x 5 cell array, type in the new formula and finally repeat the Ctrl-Shift-Enter madness.

Things get even more complicated when you don’t know the dimensions of the array up front: You might, for example, query an API or database to download time series data of unknown length and return this via your UDF.

xlwings offers a “hacky” solution for this issue: When you are on an older versions of Excel without dynamic arrays, you can use the @xw.ret(expand='table') decorator which will automatically resize the legacy array for you:

@xw.func
@xw.ret(expand='table')
def dynamic_array(r, c):
    return np.random.rand(int(r), int(c))

Still, this is hacky, slow and error prone so being able to let Excel take care of this properly is a game changer.

The new dynamic array formulas and `#Spill` errors

Whenever you use an Excel formula or xlwings UDF that returns an array, Excel will now automatically write out the whole array and put a blue line around it. No more difference to how you would type in a single cell formula and no more Ctrl-Shift-Enter! The curly braces around your formula that appear with legacy arrays are also gone.

If the array would overwrite existing cell values, it returns the new #Spill error instead, allowing you to make room for the array. Once you move or delete the offending cell, Excel returns the array automatically and gets rid of the #Spill error.

If you ever want to prevent a formula from behaving like a new dynamic array, you simply have to start the formula with an @, e.g.: =@dynamic_array(3,4) and you’re back to the old behavior.

xlwings UDFs and the new dynamic arrays

The best part about xlwings and dynamic arrays is that they work out of the box without any changes. However, if you were using the xw.ret(expand='table') decorator previously, make sure to remove it again as soon as you have upgraded to an Excel version that supports dynamic arrays!

For example, building a UDF for Quandl time series is now as easy as:

import xlwings as xw
import quandl

# quandl.ApiConfig.api_key = 'MY_QUANDL_API_KEY'

@xw.func
def get_quandl(ticker, start_date=None, end_date=None):
    return quandl.get(ticker, start_date=start_date, end_date=end_date)

And getting the Big Mac Index for the U.S. looks like that:

New built-in Excel formulas: UNIQUE etc.

Dynamic arrays come with a few new built-in Excel functions that are worth mentioning. For example =UNIQUE(A1:A12) returns a list of unique values:

Other new dynamic array formulas worth checking out are:

Tired of Excel’s VLOOKUP? Use SQL queries on Excel ranges!

2017-07-15T00:00:00+00:00

If you have the free xlwings Excel add-in installed, you can query your data using standard SQL statements, like this:

What is xlwings?

xlwings is a Python package that allows us to write Python functions instead of VBA code. It recently added a full blown Excel add-in that brought us “xlwings extensions”: These are Excel functions that are available as soon as we install the add-in without any more setup steps required. The first built-in extension is the “In-Excel SQL” extension that brings the power of SQL to your Excel spreadsheets. Typically, you would use the VLOOKUP formula to connect data from two distinct regions in Excel. Or maybe you would prefer INDEX/MATCH over VLOOKUP if you are an advanced user. However, the functionality of these formulas is limited and if you have a lot of these functions, Excel can become quite slow. Often, we were banging our heads against the wall saying “this would be so easy in SQL!”, well…now it is.

The sql formula

The usage of the sql formula is simple:

=sql("SQL Statement", table_a, table_b)

You simply have to refer to your tables as a, b etc. in your SQL statement in the order you pass them in.

For more complex queries, it is advisable to write the query into a cell instead of directly in the formula which also allows you to use multiple lines in your query (by using Alt-Enter for a new line), e.g.

Behind the scenes

xlwings uses an in-memory sqlite database to provide the functionality, so refer to their homepage for further information regarding the sqlite dialect of SQL.

Unit Tests for Microsoft Excel

2016-10-17T00:00:00+00:00

Many (business) users use Excel to create full blown applications. However, unlike applications developed by professional software developers, Excel tools fall short of most software development practices that are considered minimal standards. This introduces risks that can lead to reputational damages and substantial financial losses (remember the «London Whale»).

This blog post introduces the concept of test-driven development before it shows you how to write an automated unit test for a VBA function. All we need is Python, a free and open-source programming language and xlwings, our Python package that connects to Excel. The concepts shown work on Windows and Mac.

If you already know what I am speaking of and don’t want to go through all the step by step instructions below: Simply head over to GitHub where you can clone or download a fully working example. For a broader but less technical view of this topic, download our white paper.

Overview

The directory of horror: «John_handover»
Test-driven development
The missing piece: Python with xlwings
Before we can start: Install Python
A simple unit test
No guarantee for success
Where to go from here

The directory of horror: «John_handover»

We’ve all been there: On day one of our new job we are assigned the responsibility over a folder called «John_handover». John left the company a while ago, but not without leaving behind a spreadsheet tool of about 20 MB of size, containing VBA and complex formulas that involve multiple sheets (some of them hidden) and countless VLOOKUPs. Your boss is kind enough to tell you the secret of how to manage it: «Just don’t touch anything and you will be fine.»

After getting over your first shock and thinking about how to get out of the mess, you come up with the solution of having IT transform this Excel tool into a proper application. But it turns out that the IT budget has already been spent for this year and for next year there are already a few other projects queuing up.

Bottom line is that you’re left with a tool that causes you a headache and that you don’t really understand. You would like to simplify it but are afraid of breaking something.

Guess what: In software development, this situation is the rule rather than the exception. But there’s ways to get confidence over such legacy tools and make sure that they still work as they are supposed to after a change. Say hello to test-driven development.

Test-driven development

A core component of modern software development is writing automated tests. Most commonly, these tests make sure that a small independent unit of the program’s overall logic works correctly given a few test cases that are therefore called «unit tests». The paradigm that wants you to write the tests first and only then adopt the logic of the program until all tests pass, is called «test driven development» (TDD). In a nutshell, the thinking behind TDD is: «If it’s not tested, it’s broken» (Bruce Eckel in his book «Thinking in Java», 3rd edition).

Unit tests are a very effective way to automatically test a large number of use cases (including edge cases) to make sure that a program still works correctly under all tested circumstances even after adding or changing pieces of the program.

Practically all common programming languages come with either a built- in framework for writing unit tests or integrate easily with an existing framework. Excel, however, doesn’t offer any means to effectively write unit tests. That is, until now.

The missing piece: Python with xlwings

Python is a widely used programming language that excels at interacting with all kinds of different systems which is why it is sometimes referred to as a «glue language». To interact smoothly between Excel and Python, we created xlwings, a Python package that is free and open-source, like Python itself.

xlwings allows you to automate and program Excel with Python and therefore gives you access to Python’s built-in unittest module. Writing unit tests for Excel with Python and xlwings is simple and intuitive:

No Excel add-ins required
No changes to the original Excel file necessary
Create tests covering spreadsheet logic or VBA code
You only need a Python installation with xlwings

Before we can start: Install Python

Simply download Anaconda, a free scientific Python distribution that comes in the form of a one-click installer that already contains everything we need. After downloading, install Anaconda with the defaults.

If it worked out correctly, you should see something like the following when typing python at a command prompt (on Windows) or a Terminal (on Mac):

C:\Users\felix> python
Python 3.5.2 |Anaconda 4.2.0 (64-bit)| (default, Jul  5 2016, 11:41:13) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

A simple unit test

Create a new macro-enabled workbook with the name mybook.xlsm and save it in the folder where you started the command prompt: in my example that would be C:\Users\Felix. Open the VBA editor by clicking Alt-F11, Insert a new VBA module (Insert > Module) and paste the following simple VBA function:

Function mysum(x, y) As Double
    mysum = x + y
End Function

To access this function in Python, you can easily map it with xlwings: In the Python session we started above, execute the following lines to import the xlwings package, then map the workbook and finally the VBA function. If you have saved your workbook somewhere else than in the current working directory of the command prompt you would need to provide its full path.

>>> import xlwings as xw
>>> wb = xw.Book('mybook.xlsm')
>>> mysum = wb.macro('mysum')

Now you can use mysum as if it was a native Python function but it’s actually running the VBA code:

>>> mysum(1, 2)
3

Let’s now create a Python file called test_mybook.py. In your favorite text editor, copy/paste the following into that file:

import unittest
import xlwings as xw

class TestMyBook(unittest.TestCase):
    def setUp(self):
        # setUp will be called before the execution of each unit test
        self.wb = xw.Book('mybook.xlsm')  # map workbook
        self.mysum = self.wb.macro('Module1.mysum')  # map function

    def test_mysum(self):
        result = self.mysum(1, 2)  # get result from VBA
        self.assertAlmostEqual(3, result)  # test if result corresponds to the expected value

if __name__ == '__main__':
    # This allows us to easily run the tests from the command prompt
    unittest.main()

This file contains quite a bit of boilerplate code but the actual unit test is defined in test_mysum where we assert that the result as delivered by the VBA function corresponds to the expected result. We use assertAlmostEqual so that the test doesn’t fail because of floating point issues.

To run all unit tests in that file (ok, it’s just one for now), simply call the following from a command prompt (if you still have a Python session running, then make sure to quit first by hitting Ctrl-D or entering quit()):

C:\Users\felix> python test_mybook.py 
.
----------------------------------------------------------------------
Ran 1 test in 0.083s

OK

The report gives you a dot for all tests that ran successfully. If we would introduce an error in the VBA code by changing it like this:

Function mysum(x, y) As Double
    mysum = x * y
End Function

Then the output would accordingly tell us about the failed test like this:

C:\Users\felix> python test_mybook.py 
F
======================================================================
FAIL: test_mysum (__main__.TestMyBook)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_mybook.py", line 17, in test_mysum
    self.assertAlmostEqual(3, result)
AssertionError: 3 != 2.0 within 7 places

----------------------------------------------------------------------
Ran 1 test in 0.093s

FAILED (failures=1)

No guarantee for success

Unit tests are tremendously helpful in regaining trust over a complex workbook and feel comfortable before doing changes. However, even if all tests pass, it’s in no way a guarantee for bug free code.

First, only situations that you can think of are tested and second, the tests might pass if you pick the wrong edge cases. In the sample above, if I would have tested 2 + 2 = 4, instead of 1 + 2 = 3, then the unit test would have still passed when using the multiplication by mistake. Hence it usually makes sense to test a few scenarios including various edge cases.

Where to go from here

The unit test that we’ve built in this blog post is just a simple example of writing a test that checks the logic of a VBA function.

However, you can use the same unit testing infrastructure to validate a model by comparing the output of a VBA function with the output of an alternative implementation, e.g. in one of Python’s heavily tested 3rd party packages.

Also, you can write tests that control the correct formulas and relationships in cells as this is where you often introduce errors by copy/pasting or by dragging the wrong cells and consequently filling cells with the wrong formula.

To get an overview over these techniques, too, download our white paper.

How to fetch Market Data in Excel like a Pro

2016-09-23T00:00:00+00:00

Get market data from Quandl and other data providers directly via cell formulas:

Overview

Bloomberg’s infamous BDH formula
Enter xlwings
Building QDH

Bloomberg’s infamous BDH formula

Tell a financial analyst to download market data into Excel, and the odds are that they will start typing the following formula in an Excel spreadsheet: =BDH("TICKER", "PX_LAST",...). This formula stands for “Bloomberg Data History” and will download the end-of-period prices of the indicated ticker directly into your spreadsheet - given that you have Bloomberg’s Excel addin connected to a Bloomberg terminal.

Now, the special thing about that formula is that it fills your sheet with an array of historical prices from a single cell - without being a proper Excel array formula! While this is slightly against the idea of how Excel is supposed to work, it’s very useful in practice and usually one of the first things you start missing when you change jobs and loose access to a Bloomberg terminal.

Enter xlwings

xlwings is a free and open-source Python package that allows you to ditch VBA in favor of Python and therefore allows you to achieve quite a few nice things quite easily. One of these nice things that we’ve added with the recent release of v0.10 is the possibility to create BDH-style formulas for any available data source and in exactly the way you like!

Building QDH

While it’s equally easy to use Yahoo! Finance or Google Finance data (e.g. by using the pandas-datareader package), for this sample, we’re going to use Quandl, a data platform for both free and paid data. Although Quandl offers an excellent Excel addin, it’s missing (you guessed it) a BDH-like formula. Let’s build it and call it “QDH”.

Step 1: Install Python with xlwings

While there are many ways to install Python and the required dependencies, we strongly recommend you to download and install Anaconda which already comes with everything we need in a single installer.

Just run conda upgrade xlwings from a command prompt to make sure you have at least version. Please follow the xlwings docs to install the xlwings addin.

Step 2: Create the QDH formula

Let’s start a new xlwings project by executing the following from the command prompt:

xlwings quickstart qdh

This will create a qdh directory with an Excel file qdh.xlsm and an empty Python source file qdh.py. In the Python source file we’re now going to write the following function:

import xlwings as xw
import quandl

quandl.ApiConfig.api_key = 'MY_QUANDL_API_KEY'

@xw.func
@xw.ret(expand='table')  # this makes it a dynamic array
def QDH(ticker, start_date=None, end_date=None):
    return quandl.get(ticker, start_date=start_date, end_date=end_date)

Note: You don’t need to provide a Quandl API key (comes with a free account), but the number of calls will be limited, see here.

That’s all we need for the most basic version of QDH! The magic happens in the return decorator where we tell the formula to autoexpand the data into a dynamic array.

Switch over to Excel, import the function via the xlwings addin, type =BDH("WIKI/AAPL", "2016-01-01") into a cell and hit enter. You will be presented with the end-of-day information for Apple since the beginning of 2016 (this corresponds to the animated gif at the top of this article.)

We can make this formula a bit more useful and robust by changing it like this:

import xlwings as xw
import quandl
import datetime as dt

quandl.ApiConfig.api_key = 'MY_QUANDL_API_KEY'

@xw.func
@xw.ret(expand='table')
def QDH(ticker, column, start_date=None, end_date=None):
    if isinstance(start_date, dt.datetime):
        start_date = start_date.strftime('%Y-%m-%d')
    if isinstance(end_date, dt.datetime):
        end_date = end_date.strftime('%Y-%m-%d')
    return quandl.get('{0}.{1}'.format(ticker, column),
                      start_date=start_date, end_date=end_date)

Now we can link to date-formatted cells and also specify which column we would like to fetch. Following the setup of the animated GIF, type the following into cell A4: =QDH(A3,11,$B$1,$D$1). Here, 11 is the 11th column from before, i.e. the Adj. Close:

Python in Excel alternative: Open. Self-hosted. No limits.

My thoughts on Python in Excel

Table of Contents

Make something people want

The mistake: placing the Jupyter notebooks cells inside the grid

It looks like a duck, but it ain’t one

Do repeat yourself

Magic, magic everywhere!

Non-interactive data analysis

Diagnostics pa(i)ne

Python in Excel the Cloud

A glue language without glue

Matplotlib is great, but a little interactivity wouldn’t hurt

No access to the Excel object model

DataFrames are great, but so are NumPy arrays and lists

A better alternative than “an alternative to the Excel formula language”

What are my plans with xlwings?

Conclusion

A New Way to Automate Google Sheets with Python

Table of Contents

How does the traditional Sheets API work?

The xlwings quickstart for Google Sheets

Authentication and authorization

Security

Run your code via buttons, custom menus, and triggers

DataFrames and plots

Production deployment

Roadmap

Conclusion

What Makes Excel’s Lambda Functions so Awesome (and what doesn’t)?

Table of Contents

Why Do We Need Lambda Functions?

Create your First Lambda Function!

How Do Lambda Functions Work?

Why are Lambda Functions so Awesome

Limitations with Lambda Functions

Managing Lambda Functions Across Workbooks

Keeping Track of Your Lambda Functions

Conclusion

5 Python Libraries for Reporting and Factsheets

Table of Contents

Overview

Pandas

Pandas + Excel

Pandas + HTML

xlwings

xlwings (Open Source)

xlwings PRO

Plotly Dash

Alternatives to Plotly Dash

Datapane

Datapane (open-source library)

Hosted Reporting Apps

ReportLab

ReportLab OpenSource

ReportLab PLUS

Conclusion

Dynamic arrays in Excel with Python and xlwings

Overview

What’s wrong with the legacy Ctrl-Shift-Enter array formulas?

The new dynamic array formulas and #Spill errors

xlwings UDFs and the new dynamic arrays

New built-in Excel formulas: UNIQUE etc.

Tired of Excel’s VLOOKUP? Use SQL queries on Excel ranges!

Overview

What is xlwings?

The sql formula

Behind the scenes

Unit Tests for Microsoft Excel

Overview

The directory of horror: «John_handover»

Test-driven development

The missing piece: Python with xlwings

Before we can start: Install Python

A simple unit test

No guarantee for success

Where to go from here

How to fetch Market Data in Excel like a Pro

Overview

Bloomberg’s infamous BDH formula

The new dynamic array formulas and `#Spill` errors