SPFx overclockers or how to significantly speed up the "gulp serve" command

A few months ago I wrote an article about SharePoint Framework build performance - SPFx overclockers or how to significantly improve your SharePoint Framework build performance. I've tried to reduce the amount of time the "gulp serve" command uses to re-build your code and to finally refresh a browser. I used different optimization technics for that purpose. The idea was to tweak the default SPFx build pipeline. However, those post only partially solves the problem. 

In this post, I will solve the problem from another way around (spoiler: I managed to make "serve" 10-15 times faster). 

The idea

How SharePoint Framework's "gulp serve" works? It gets your sources and outputs javascript bundles. But "gulp serve" becomes slow when your solution grows. How to fix that issue? Well, since it's slow, then don't use it! 

So this is the idea - don't use "gulp serve", use completely custom webpack based build to transform your sources into exactly the same javascript bundles which are produced by "gulp serve". 

What is the input for "gulp serve"? - TypeScript sources, styles. What is the output? - Javascript files. Can we get sources and produce exactly the same javascript? -Yes.

How it works

Some nerdy content goes below. If you don't want to read about internal implementation, go directly to "How can I use it?"

To make it work, we need custom webpack config and webpack dev server to serve webpack output. 

webpack config

The most essential part of the webpack config is below:

module: {
    rules: [
      {
        test: /\.tsx?$/,
        loader: 'ts-loader',
        options: {
          transpileOnly: true,
          compilerOptions: {
            declarationMap: false
          }
        },
        exclude: /node_modules/
      },
      {
        use: [{
          loader: "@microsoft/loader-cased-file",
          options: {
            name: "[name:lower]_[hash].[ext]"
          }
        }],
        test: /.(jpg|png|woff|eot|ttf|svg|gif|dds)((\\?|\\#).+)?$/
      },
      {
        use: [{
          loader: "html-loader"
        }],
        test: /\.html$/
      },
      {
        test: /\.css$/,
        use: [
          {
            loader: "@microsoft/loader-load-themed-styles",
            options: {
              async: true
            }
          },
          {
            loader: 'css-loader'
          }
        ]
      },
      {
        test: /\.scss$/,
        use: [
          {
            loader: "@microsoft/loader-load-themed-styles",
            options: {
              async: true
            }
          },
          'css-modules-typescript-loader',
          {
            loader: 'css-loader',
            options: {
              modules: true
            }
          }, // translates CSS into CommonJS
          "sass-loader" // compiles Sass to CSS, using Node Sass by default
        ]
      }
    ]
  },
  plugins: [
    new ForkTsCheckerWebpackPlugin({
      tslint: true
    }),
    new webpack.DefinePlugin({
      'process.env.NODE_ENV': JSON.stringify(process.env.NODE_ENV),
      'process.env.DEBUG': JSON.stringify(true),
      'DEBUG': JSON.stringify(true)
    })],

The key things here are loaders:

  • ts-loader transforms TypeScript to Javascript. It doesn't perform type checking (transpileOnly=true), thus works very fast. More on that below in plugins section
  • @microsoft/loader-cased-file - that's the same loader used by SharePoint Framework webpack task, it loads fonts, images
  • html-loader - loads HTML files. It's included into the default SharePoint Framework webpack task, so I added it as well
  • @microsoft/loader-load-themed-styles - that's also OOB loader, it transforms themable css into real css colors, thus very important
  • sass-loader - loads .scss files and transforms to .css
  • css-modules-typescript-loader - this loader transforms imports from *.module.scss files into TypeScript type definitions. 

Also, I use ForkTsCheckerWebpackPlugin. It performs type checks and listing in a separate asynchronous thread, thus it doesn't affect build performance. 

But it's not all. We also need "externals", "output" and "entry" values for our webpack config. Basically, these values are based on your SharePoint Framework solution configuration. And here is the trick - I have a pre-build task, which writes intermediate SPFx webpack config to the disk. Than when custom webpack build starts, it reads that config and sets correct "externals", "output" and "entry" values. 

webpack dev server

 We also need to serve our bundles. Webpack dev server is a perfect solution for that task:

devServer: {
    hot: false,
    contentBase: resolve(__dirname),
    publicPath: host + "/dist/",
    host: "localhost",
    port: 4321,
    disableHostCheck: true,
    historyApiFallback: true,
    open: true,
    openPage: host + "/temp/workbench.html",
    stats: {
      colors: true,
      chunks: false,
      "errors-only": true
    },
    proxy: { // url re-write for resources to be served directly from src folder
      "/lib/**/loc/*.js": {
        target: host,
        pathRewrite: { '^/lib': '/src' },
        secure: false
      }
    },
    headers: {
      'Access-Control-Allow-Origin': '*',
    },
    https: {
      cert: CertificateStore.instance.certificateData,
      key: CertificateStore.instance.keyData
    }
  },

Some key things here:

  • publicPath should be "https://localhost:4321/dist", since that's the location used by OOB "gulp serve"
  • I use proxy to re-map requests for localization resources from lib folder to src
  • for https configuration, I use OOB certificates (@microsoft/gulp-core-build-serve/lib/CertificateStore)

Customized gulpfile.js

In gulpfile.js we need two things - write intermediate webpack config to disk and ensure that workbench is created. It's possible to do that via below tasks:

const argv = build.rig.getYargs().argv;
const useCustomServe = argv['custom-serve'];
const fs = require("fs");
const workbenchApi = require("@microsoft/sp-webpart-workbench/lib/api");

if (useCustomServe) {
  
  const ensureWorkbenchSubtask = build.subTask('ensure-workbench-task', function (gulp, buildOptions, done) {
    this.log('Creating workbench.html file...');
    try {
      workbenchApi.default["/workbench"]();
    } catch (e) { }

    done();
  });

  build.rig.addPostBuildTask(build.task('ensure-workbench', ensureWorkbenchSubtask));

  build.configureWebpack.mergeConfig({
    additionalConfiguration: (generatedConfiguration) => {
      fs.writeFileSync("./temp/_webpack_config.json", JSON.stringify(generatedConfiguration, null, 2));
      return generatedConfiguration;
    }
  });

}

build.initialize(require('gulp'));

How can I use it?

To make it work, you need custom webpack.js file and patched gulpfile.js

To make your life easier, I've created a CLI tool, which does all the job - spfx-fast-serve. These are the required steps:

  1. npm install spfx-fast-serve -g
  2. Open a command line in a folder with your SharePoint Framework solution you want to speed up.
  3. Run spfx-fast-serve and follow instructions. In most cases, you shouldn't do anything specific and the cli "just works".
  4. Run npm install
  5. Run npm run serve and enjoy the incredible speed of serve command!

Which SharePoint Framework versions are supported?

The latest version of the tool supports the latest SPFx for SharePoint Online. 

SharePoint 2019 is also supported - but you need to run specific version via npx:

npx spfx-fast-serve@1.4.x

SharePoint 2016 is NOT supported.

Testing performance gain with different projects

I've tested the tool with various projects, including open-source ones. Check out the table below to see the difference in refresh time (the time needed to compile your project when you change a file and start refreshing a page in a browser) between gulp serve and spfx-fast-serve:

  gulp serve spfx-fast-serve
The default "Hello World"
React web part
3-5 sec 0.3-0.5 sec
PnP Modern Search solution 28-34 sec 2-4 sec
SP Starter Kit solution (v1) 40-50 sec 2-3 sec

NOTE: The actual time depends on the environment, hardware, but at least you can see the difference

The performance gain is significant for all types of projects. 

Final thoughts

spfx-fast-serve has below features/advantages over the default gulp serve:

  • works significantly faster
  • all compilation are done in a memory with webpack, no additional "copy", "prepare", "typescript", "whatever" tasks.
  • incremental TypeScript compilation when a file is being changed. It means only necessary files are compiled, not everything.
  • asynchronous type checking and linting.
  • supports local and hosted workbench
  • live reloading (for hosted workbench as well)
  • doesn't mess up your default SPFx build. If you have troubles, simply switch back to regular gulp serve

Please try it in your SharePoint Framework solution and tell me if it works for you. 

What’s new and what’s changed in SharePoint Online REST API in August-October 2019

I continue exploring REST API changes using my tool called SharePoint REST API Metadata Explorer. Here I post only some interesting findings, the full log is available for you on the SharePoint REST API Metadata Explorer web site under the "API Changelog" tab. 

Disclaimer

Please note, that all changes are gathered from Targeted tenant. Most likely these changes haven’t been officially introduced yet, use this post as spoilers to potential upcoming features. If you want to use APIs mentioned here in production, please check corresponding official documentation to make sure they are available.

AI, Machine Learning and Knowledge hub

The previous week was "Ignite" week. A lot of new things were introduced. Including innovations in the AI area. More...

Exploring The Microsoft Graph Toolkit

At the end of September 2019, Microsoft released The Graph Toolkit library.

a collection of reusable, framework-agnostic web UI components that work automatically with Microsoft Graph

It was in preview for a while, now it's in GA, thus it's a good time to start exploring what is available in this library.  

First of all, that's a set of components, which abstract a lot of things about MS Graph, authentication, UI away from a developer. It provides a very seamless interface for building UI components. UI elements are built with web components technology - which means that they are framework agnostic. You can use it with any modern JavaScript framework.

Let's explore how to use this library with a React-based single page application and with SharePoint Framework. More...

What’s new and what’s changed in SharePoint Online REST API in March-April 2019

What's new and what's changed in SharePoint REST API in March in April 2019? Explore the most interesting changes (additions and updates) here! 

Just a quick reminder, all data come from my SharePoint REST API Metadata Explorer. Go to the "API Changelog" tab and see what's changed in recent months in SharePoint REST API. 

Disclaimer

Please note, that all changes are gathered from Targeted tenant. Most likely these changes haven’t been officially introduced yet, use this post as spoilers to potential upcoming features. If you want to use APIs mentioned here in production, please check corresponding official documentation to make sure they are available.

Organizational News (aka authoritative news)

In the previous episode, I mentioned that OrgNews endpoint was added. Now, when corresponding PowerShell cmdlets (Get-SPOOrgNewsSite, Set-SPOOrgNewsSite, Remove-SPOOrgNewsSite) to manage Organizational news sources (also known as authoritative news) were announced, you can also use REST API to list all organizational news sources. However, I haven't found a way to add or remove authoritative news sources using the REST API. More...

What’s new and what’s changed in SharePoint Online REST API in January-February 2019

I haven't posted about changes in REST API for a while, because nothing interesting was really happening inside REST API in December-November. But now we have something new to explore! 

Just a quick reminder, all data are coming from my SharePoint REST API Metadata Explorer. Go to the "API Changelog" tab and see what's changed in recent months in SharePoint REST API. 

Disclaimer

Please note, that all changes are gathered from Targeted tenant. Most likely these changes haven’t been officially introduced yet, use this post as spoilers to potential upcoming features. If you want to use APIs mentioned here in production, please check corresponding official documentation to make sure they are available.

Organizational News

This item was in the roadmap for a while. You will be able to "mark" a site as "organizational news source". When viewing news from "organizational news source" you will see an indication that this news is "organization news". Nice feature to distinguish global company news from department news. More...

Introducing SharePoint Typed Item - Visual Studio Code extension

When working with SharePoint as a developer you often need access to SharePoint data. If you use TypeScript you expect that all data will be strongly typed. However, it's not a case if you work with SharePoint dynamic data like lists and libraries. 

Consider the code below written with PnPjs:

sp.web.lists.getByTitle('Clients').items.get()
      .then((items: any[]) => {
        for (const item of items) {
          console.log(item);
        }
      });

or with SPHttpClient inside SharePoint Framework solution: 

this.context.spHttpClient.get(`${currentWebUrl}/_api/web/lists/getByTitle('Clients')/items`, SPHttpClient.configurations.v1)
      .then((response: SPHttpClientResponse) => {
        response.json().then((items: any[]) => {
          for (const item of items) {
            console.log(item);
          }
        });
      });

In both cases, you don't know the type of items beforehand and have to use any type (items: any[]). What's wrong with any type? In TypeScript it's considered as bad practice. Of course you can and should use any in some edge cases, but in general, it's considered as bad. With any type you don't have type intellisense in vscode, you lose type checking if you want to use your variable elsewhere. So how to fix it? 

To fix it you should create a separate interface manually and describe all needed fields. In that case instead of any you use: 

items: Client[]

While it works, it has a few drawbacks: 

  • you have to create all needed interfaces manually
  • if you have a need to access additional fields, you will have to go and update corresponding interfaces
  • if you (or team members) add a field to a list, you will have to go and update corresponding interfaces

What if there is a tool, which generates all required always up-to-date interfaces for you?

This is exactly what SharePoint Typed Item does - it takes your configuration and outputs TypeScript interfaces based on SharePoint lists, libraries and content types. 

Read further to find out how to get started with SharePoint Typed Item extension. More...

Building interactive feedback analysis system with MS Forms, MS Flow, Power BI and SharePoint Online

Office365 family provides a lot of services for you and your organization. In recent weeks I was playing \ working with some of them like Power BI and MS Flow.

I wrote an article about sharepoint.stackexchange analysis with Power BI. Power BI is a great tool for data analysis. With Power BI Desktop you can create great visualizations of your data and share it with your colleagues or publish on the web. It nicely integrates with modern SharePoint Online as well. 

MS Flow provides a way to automate a lot of different processes and simplify business cases. It also a very powerful tool in a different area - process automation. How to combine all of them and build something useful, engaging and interesting? That's the question I came up when I was working with these tools. Well, there are a lot of options available. It all depends on your imagination or concrete business case. I ended up with an interactive feedback analysis system. More...

SharePoint development state in 2018: story based on sharepoint.stackexchange analysis with Power BI

The year 2018 is over and it's time to perform regular analysis of data at sharepoint.stackexchange. This is the third edition of such analysis. 

Tools used to collect and analyze data: 

  • Power BI with Power BI Desktop - super cool tools for data analysis. If you don't have experience with Power BI, it's worth trying to see what is possible. When I first tried it two years ago I was sooo impressed with power yet simplicity in performing data analysis and building visualizations. It works very well for both simple and advanced scenarios. I believe that everybody will find these tools useful for any kind of data analysis. 
  • DaxStudio - extremely useful tool to test your DAX queries. I found it recently and it helped me a lot.
  • Power BI Community - Power BI has a very strong community. I found a lot of answers at their forum, I even asked some questions and community helped with valid answers. That's not a "tool" but worth mentioning. I am grateful for all the answers.
  • Google Maps Geocode API
  • Stack Exchange API
  • osmosis - nodejs webpages scrapper

The source code used to gather initial data is available at GitHub

Links

Previous reports:

Disclaimer

All thoughts are mine. Maybe they are not correct or you simply think differently. Please share your thoughts and opinions in comments. 

Ok, let's get started! :)

More...